Browser Tools and Testing WG, day 1 -- 25 Oct 2018

<ato> ‘allo ‘allo

<ato> RRSAgent: make minutes

<ato> RRSAgent: Please make logs public

<ato> RRSAgent: Please make minutes

<ato> RRSAgent: Meeting title: Browser Tools and Testing WG, day 1

<ato> RRSAgent: Meeting: Browser Tools and Testing WG, day 1

<ato> RRSAgent: listen

<ato> RRSAgent: make minutes

Folks, the agenda is here: https://www.w3.org/wiki/WebDriver/2018-TPAC-F2F

<jgraham> RRSAgent: make minutes

<scribe> scribe: simonstewart

<AutomatedTester> https://www.w3.org/wiki/WebDriver/2018-TPAC-F2F

<scribe> scribenick: simonstewart

<jgraham> RRSAgent: make minutes

AutomatedTester starts with introductions

AutomatedTester: we'll then go into the state of the union, then make the agenda from the topic bank at https://www.w3.org/wiki/WebDriver/2018-TPAC-F2F
... works at mozilla

JohnJansen: works at MS on MS WebDriver "and other testing tools"

jgraham: mozillian. Works on testing stuff

ato: Also mozillian
... has worked on _everything_

whimboo: also a mozillian, works on geckodriver

MikeSmith: works for the w3 on platform test work. Encouraging everyone to go to Japan

corevo: works at applitools on selenium ide

calebrouleau: works on chromedriver. works with google developers who need to use this stuff

johnchen: also works on chromedriver at google.

ChristianBromann: works in dev tools team at saucelabs.

ThomasBoyles: works at sauce labs too

BrendynA: works on MS Edge

developer tools

AutomatedTester: next point of order is the state of the union.
... The spec has had a lot of bug fixes since we reached REC, mainly as people implement it. The spec is still a living document
... current test suite is quite extensive, but can always grow. All in WPT now. Mozilla are trying to keep those up to date so everyone can benefit
... one of the things I think is useful for the next two days is to determine the direction this group wants to go in. Some of it depends on where people are in their implementations
... shares conformance numbers. "About 90% pass rate"

JohnJansen: MS WebDriver update. Impressive pass rate. Not going to implement Proxy capabilities, where there are lots of the failures

<ato> Test results on wpt.fyi: https://wpt.fyi/results/webdriver/tests?label=experimental

JohnJansen: MS WebDriver is pretty robust

jgraham: finds it amusing how no-one knows how many tests there are in WPT

discussion ensues

brrian: status update on safaridriver
... safari 12 shipped with w3c support enabled by default. 76%ish pass rate. Bug fixes landing each week

johnchen: chromedriver working on w3c compliance. Have a big chunk missing (actions api) Still have low numbers on WPT, but partly because latest chromedriver not in test harness
... also working on corner cases. Expect by end of year there should be w3c compliant chromedriver
... explains the difficulty of synching chromedriver and chrome releases. Which version of which for what?

calebrouleau: talks about how version numbers will work moving forward (first three numbers of chrome will be chromedriver release number)

Discussion about the internals of how WPT are run

AutomatedTester: next item: setting the agenda

Setting the Agenda

AutomatedTester: first item is bi-directional communication. Also sets tone for the next few days, as many features depend on the outcome of that conversation

General lack of ability to use pens is displayed by people in the room

Discussion about whether to go for a prioritised list or time blocks

ato: points out that some of the items in the topic bank are bug fixes, but some are new features.
... a couple that are plainly informative

AutomatedTester: Possibly do "New commands for enhanced browser automation (DevTools)" after that?

ato: what is that?

ChristianBromann: that's the same thing as the first item on the agenda

simonstewart: do we want something easy after bi-di, or move on to logging?

AutomatedTester: logging

People bemoan lack of coffee

jgraham: points out that some agenda items are things we've spoken about before and come to agreement on.

ato: we need consensus on various items, since we're not great at following up on things.
... we have a few implementations are good. IEDriver and geckodriver are both very conformant.

simonstewart: we don't know how large the bi-di stuff will work

thomasboyles_: wants to discuss async new session and command pipelining

https://www.w3.org/wiki/WebDriver/2018-TPAC-F2F

ato asks who's not here tomorrow

BrendynA: is only here today, and is keen to cover bi-di (cdp)

jgraham: if we don't have cross group meetings, and don't have time constraints, we should run the topics as a queue

johnchen: so no formal agenda, but start with cdp

JohnJansen: said that, not johnchen

AutomatedTester: we'll probably finish around 5

ato: we have the room until 6

jgraham and AutomatedTester highlight how much fun we have in spec meetings

calebrouleau: asks if we start at 9 tomorrow. Answer is "yes"

AutomatedTester: suggests doing easy things in the 21 minutes until coffee

general agreement

Documenting WebDriver on MDN

ato: mostly informational. since webdriver is now a standard, it makes sense to have documentation on MDN.

<AutomatedTester> https://developer.mozilla.org/en-US/docs/Web/WebDriver

ato: someone in MS (maybe?) has already added something to `naviagtor.webdriver`. ato then linked that to the docs

(the docs being: https://developer.mozilla.org/en-US/docs/Web/WebDriver)

<ato> https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities

ato: for geckodriver mozilla have started referring to this for areas that are documented
... there's docs missing for a lot of commands and area. Plan is to have a page per item
... there's a brower compat section on the front page. BCD (Browser Compat Data) used as source for this

<simonste_> ato: it's very red right now, but that doesn't necessarily represent the truth since that data looks out of date

ato: it's very red right now, but that doesn't necessarily represent the truth since that data looks out of date

<ato> https://github.com/mdn/browser-compat-data

calebrouleau: what is bcd?

ato links to it (above)

ChristianBromann: should that table contain appium commands?

ato: appium isn't implemented by the browser vendors and isn't part of a standard

brrian: what's the guidance for non-standard things
... we have separate docs for those things.
... all the extensions are documented in a man page and in other places

jgraham: mdn has docs on things that relate to the web platform but are non-standard

brrian: is the intent for this to be the canonical source of docs for the spec, or for anyone using the spec?

ato: linking to the spec itself is a Really Bad Idea because the spec isn't written for users
... talks about Jonathan Lipps "Layman's Version of the Spec"

<ato> https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities/firefoxOptions

jgraham: mdn is more useful for users if all the information is located there, and I think it supports you documenting non-standard stuff

ato: dom element scroll into view has some options that only supported on some browsers. Those options are highlighted.
... imagines we'll do something similar for the webdriver mdn docs
... if we want this to be useful for end users, the bindings put no constraints on how people implement this

jgraham: mdn is useful for people implementing bindings

brrian: bindings docs should contain some of this information anyway

ato: already finding this useful. eg. when working on set timeouts

simonstewart: paraphrases ChristianBromann: do we include endpoints that aren't standardised such as those used by appium?

ato: I think that's quite hard to swing.

ChristianBromann: proposes at the end of tomorrow a discussion about whether to standardise appium

ato: there's nothing to stop us linking to external documentation.

<ato> https://developer.mozilla.org/en-US/docs/Web/WebDriver/Errors/InsecureCertificate

ato: really helpful for when users run into random errors (insecure certificate, for example). MDN rates highly on search, and that provides helpful, actionable advie

jgraham: documenting non-standard capabilities for standard commands might work

ChristianBromann: appium has good docs

brrian: thinks of appium docs as binding docs

ato: the reason I wanted to bring this up is that Mozilla have started writing the docs, and want to encourage folks to contribute

JohnJansen: what's the best way to contribute? Just edit the page

Mozillians nod their heads

JohnJansen: there are no PRs?

ato: bcd takes PRs
... the wiki pages: just log in and edit.

jgraham: mdn may not have unrestricted sign up.

session breaks for 30 minutes

<boaz> scribe: boaz

Chrome Devtools Protocol

AutomatedTester, we are here to talk about a bidirectional protocol

BrendynA, or hopefully devtools protocol

<JohnJansen> RRSAGENT: make minutes

AutomatedTester, there is currently a wicg working on a standard

<BrendynA> DevTools WICG URL: https://github.com/WICG/devtools-protocol

<JohnJansen> RRSAGENT: make logs public

<BrendynA> *DevTools Protocol WICG URL: https://github.com/WICG/devtools-protocol

DevTools WICG URL: https://github.com/WICG/devtools-protocol

AutomatedTester, the reason why this has come up is so that people can subscribe to events in their automated testing

simonstewart, yes its a push oriented protocol at the moment

jgraham, empirically what we have seen is... well pupeteer is the example. there are people coming up with tools based on cdp that developers like, and they are only possible because of bidirectionality.

jgraham, but the issue is its chrome only.

jgraham, but this all means that there is need identified in the market

BrendynA, its not chrome only, its edge, too

jgraham, but it cannot run pupeteer

simonstewart, there is a desire to register with a browser and find out when things change slate

jgraham, and its not a clean slate, DevTools Protocol has precedence. edge is implementing it and cares about using it for automation. gecko has not implemented, but we see people want it, so we think the next step is to adopt sdp in a direct way.

<jgraham> boaz: Wnt to add that there's more than wanting to register events, there's also an overlap between CDP and the developer tools, which devs find nice

<jgraham> boaz: Overlap is a feature for web devs

<scribe> scribe: boaz

ato, this room has mostly talked about webdriver for a long time. i suppose that the reason we are talking about this is that web developers want these features, and we are in a situation where web developers are using pupeteer, and that means they are not testing in firefox.

ato, and we have to remember that webdriver was written 15 years ago. and the world was very different. and I dont want webdriver to be seen as a competitor to cdp. web driver is a superset of what you can do with DevTools Protocol. all of the browser vendors around this table (4) have invested in this standard. we have millions of tests dependant on webdriver and it would be a problem for mozilla if it went away.

ato, I think the concern from mozilla's side is that we don't want to implement something controlled by one browser.

ato, last year we saw people implementing Devtools Protocol in servo. the concern james had a year ago when we saw that work happening is that it is easy to add the endpoints and say that hopefully that will give us the functionality, but if the semantics are not defined well its difficult to implement the behavior.

ato, and the nice thing about webdriver is its semantically defined

BrendynA, we at edge actually view the devtools protocol as a superset of webdriver.

<jgraham> Oops

simonstewart, the webdriver apis were originally designed to make testing more pleasant. we have already broken out of this api in the selenium project, where we have done logging for [unrecorded]. so it is clear that the next step in testing ecosystem is some of these features that allow developers to capture events and introspect the browser.

brrian, from the safari point of view, these are different things.

brrian, we are approached by vscode and they ask us to implement devtoolsprotocol, so people can look at web views.

brrian, that is cool, but seems difficult to specify.

brrian, also I am not sure what the overlap should be. a break point would not be good for an automated test.

brrian, so I would like us to be explicit about what should be specified.

brrian, in webkit we do have a json rpc api, but it is divergant, and un specced and we don't have plans to spec it.

brrian, what we did in webkit was make a bidirectional testing interface, but it is a different one than the web inspector.

brrian, if you try and spec the debugger protocol, you will be said. for example what does stepping mean?

<jgraham> boaz: From my perspective, dtp is a superset of webdriver, as currently implemented, Bocoup are grateful fro WebDriver, don't want to view these as competitors

ChristianBromann, I can definitely see both devtools protocol live along side each other.

ChristianBromann, I can see web driver adopting the devtools protocol in a way that it only supports things that are in all web browsers.

BrendynA, two points, this could be naive, because we have eaten low hanging fruit for debugging protocols. I dont think that we need to specify the internal semantics of a debugging protocol. we have reverse engineered in/out puts in vscode and edp and tested it in vscode.

BrendynA, also, we do have multiple use cases for this; remote debugging, driving external devices, etc.

BrendynA, debugging was a strawperson for us.

we want to specify here the subset of cdp that relates to automation

JohnJansen, I think it is super important that this not get branded as a chrome thing. this is Devtools Protocol.

JohnJansen, also, I want to say with a few tweak pupeteer would work with edge's implementation.

ato, I think what people want is some kind of ubiquitous remote protocol. some people want to do automation, devtools, and some people want introspection.

ato, I think the answer tpo brrian's concern about our inability to specify some things. that is astute, and is true, but I think it would be very valuable to have one protocol to do these three things.

ato, in ff we have at least 3 different teams doing at least 3 different remote protocols.

ato, we want that to stop.

ato, and also, we need to balance the concerns here. yes there are somethings that would be different to spec and some things that would be too chrome specific. but on the other hand, web authors want to use these tools, and I am concerned that if we do not integrate them web authors will only test in chrome.

ato, and thirdly

<JohnJansen> to ato's point about Puppeteer being Chrome only: https://developers.google.com/web/tools/ there is no mention of WebDriver

ato, if we go down the path of specifying a remote protocol, I am worried that if we try make what is currently CDP more suitable for webkit and edge and gecko, that we may fall into the same trap as what happened with webdriver. 6 years later I think we're realizing that we may have changed the protocol too much.

(to ato's point about Puppeteer being Chrome only: https://developers.google.com/web/tools/ there is no mention of WebDriver)

ato, but my hope that we can get to some kind of future where we haven't changed it too much, because with webdriver and selenium it was very painful for web authors to have to follow the changes to the technology. we broke their tools.

jgraham, I share brrian's concerns that there is too much to spec, I think that trying to build something stable on top of unstable like CDP is a bad idea. I've heard tell from people who have tried to do this, that it has been hard.

jgraham, the CDP derived devtools protocol has an existence of an ecosystem that use it, which intrinsicly implies some compat constraints.

jgraham, these compat constraints likely mean that we can rely on the api not changing so much.

jgraham, I think I would be very concerned if the internal model of the world that we need to make this work does not match what chrome needs.

jgraham, I don't know to what extent, for example, the model of windows, matches the standard way that other browsers do windows

jgraham, if it does not, then I think this would be very hard to do.

jgraham, I dont have a solution, but I have a pile of worries.

calebrouleau, from the perspective of chrome driver, I am also concerned about what layer we want to standardize on. chrome DP relies on non standard internals, so I think we will need a layer on top of existing implementation.

<BrendynA> https://github.com/WICG/devtools-protocol/blob/master/meetings/agenda-tpac2018.md

BrendynA, we tried to propose an agenda for today, I will paste in a link to that.

BrendynA, I want us to "align on" this technology, thats why we chose a wicg

BrendynA, when we made chrome devtools protocol work in edge, we only had one internal system we had to change to match chrome, so there was not too much of an issue with chrome internals.

BrendynA, when you look at how the DTP is invoked, feature detection, specific http endpoints, how to handle apis, and then lastly, if we can make it through those foundational things, then are there specific apis to align around, that is secondary to our desire to have baseline architectural technology alignment.

simonstewart, cool, so, i think we're getting really focused on chrome debugging protocol, and taking a step back, I want to point out that the api we spent the last 6 yrs specifying is implemented natively in 0 apis.

simonstewart, I think what we all agree on is that we need biderectional communication, and that it should be based on what Chrome and Edge did, but that it would not be a hard requirement for browsers to use this for debugging.

simonstewart, I am hearing no disagreement on this, im hearing its too big to spec, which it is. So should we sit down and talk about use cases for what we want to spec here.

simonstewart, when I talk to people about what they like about pupeteer, I hear that they like to listen to events, spool something?, and then from a browser perspective, people like to use debugging features.

simonstewart, I think whatever we do from a webdriver perspecitve, what we spec here can be very minimal and jsut pass through to browser's to handle.

simonstewart, but I think that specifying the primitives that are there would be sensible.

brrian, i second a lot of what simon just said

brrian, i disagree that setting a break point has idomatic meaning; does it take into account sourcemanps line endings, pages that haven't loaded yet

<brrian> https://docs.microsoft.com/en-us/microsoft-edge/devtools-protocol/0.2/domains/runtime#callfunctionon

brrian, I don't think that what is there in CDP the thing we want to expose

brrian, for example runtime callfunction: https://docs.microsoft.com/en-us/microsoft-edge/devtools-protocol/0.2/domains/runtime#callfunctionon

brrian, we need to be careful about what we are exposing so that people can understand what it does.

and that is not true at all in any devtools protocols.

brrian, there are all sorts of implementer detials which we should hide as much as possible

brrian, so I think we should start with Json RPC and go from there

brrian, I also have some concern about introspection and the issue of capabilities and how does that match with a normal thing, is it an upgrade from WebDriver?

brrian, right now you have a driver and every browser hosts a web server, that is not realistic

brrian, we need to make a more sane architecture for when you put devices, emulators, or web views into the picture

BrendynA, that is what we published to the WICG

<AutomatedTester> https://github.com/WICG/devtools-protocol/

https://github.com/WICG/devtools-protocol/blob/master/meetings/agenda-tpac2018.md

BrendynA, json, websockets and what endpoints are exposed, that is what we want to spec

<ato> Scribe: ato

<BrendynA> These are the topics we are hoping to cover at this TPAC: https://github.com/WICG/devtools-protocol/tree/master/docs

<Zakim> jgraham, you wanted to say that Mozilla's intent is not to move Firefox devtools to this protocol

jgraham, So, I wanted to say something about the scope.

jgraham, We are concerned explicitly with the automation part. There is no intent to replace Firefox devtools with something CDP-derived. That is a place where there are a lot of differences between browsers. There is interest in interop with VSCode for the subset of the things it does, but I'm less concerned with the details of the use case.

jgraham, There is a relatively small number of people writing IDEs.

jgraham, For automation on the other hand, there is a lot of people writing automated test cases.

jgraham, The bits that are required to enable the use case of Puppeteer are to Mozilla the most interesting things we should define.

jgraham, From there we can expand, for example to do breakpoint setting but with no guarantees that this would behave the same in browsers.

jgraham, Regarding upgrading from WebDriver, WebDriver might provide a good selection mechanism because it allows you to match capabilities to get the right node.

jgraham, But the overall session looks like a WebDriver session.

jgraham, There are a set of useful features developed in WebDriver that are independent from the actual how-you-automate-the-browser part.

calebrouleau: Are those features used a lot?

ChristianBromann: Sauce Labs is very interested.

jgraham, I don’t know.

<boaz> scribe: boaz

ato, i thought that point that james made just now was a good point. I didnt think about how there are things unique to webdriver like matrrix selection that are important.

ato, I have de ja vu sitting here. I used to work at opera, and 12 years ago, we came here to tpac and sat around a similar table and asked other browser implementers to stanfdardize on something like cdp, it was called fire somthing.

ato, I am happy that we are finally back at this table.

ato, one of the great things I hear about cdp, is that has domains and scopes, (ironic because scopes was the name of our protocol 12 years ago), it seems like there are many things that can be shared. there are a subset we need for the pupeteer use case, but then there are vendor only things.

ato, I met ChristianBromann in chicago a few days ago, and you told me you were working on a webdriver client that implements webdriver but also implements CDP.

(DP)

simonstewart, I have a java cake? that does the same

ato, so we are seeing examples of over laps in the wild

ato, one of the unique things about webdriver is that the transport is not defined

brrian, this is good because I would not be able to implement a web server in a browser

ato, from my point of view, having somehting in web driver like get and point which give you a web socket and let you go into cdp would be good

ato, and if browsers want to expose a tcp server, thats fine

sah, hi im from sauce labs

sah, theres a lot of stuff that is in devtools that is not necessarily automation, but is still useful for testing

sah, also break points are useful for test authoring...

sah, so as we think about what are important features, things more than pupeteer could be usefule

sah, second, in the testing world there are lots of companies with large test farms or they use cloud services like sauce labs. so it would nice to take into account ways to take deal with proxies, and think about scaling

<ato> Scribe: ato

ChristianBromann, We build custom commands for throtteling the network. Also allows users to intercept requests that are made. We have found that it has had a great adoption and it is core in our users tests.

boaz, I just want to zoom out. It is reasonable for the platform to standardise debugging APIs. It is reasonable, and programming languages and environments have done this in the past. This is a reasonable thing to talk about in a W3C context. I don't think it's great for the web, or for web developer, that things like devtools is part of the browser

product.

boaz, product

boaz, There is some view that devtools is product differentiators for vendors. As a web author it would be great if they were standards.

BrendynA, On the point of not wanting to standardise the way to initialise the protocol, that seems reasonable.

BrendynA, There should be a common understanding of how users can initialise the connection. Specifying things that are standard APIs vs. things that are specialised for certain browsers. Versioning of these services. For the sake of implementors—from VSCode's view—that there is a reliable way to get at it.

BrendynA, We also have a range of devices that we want to interact with, and there are constraints that are unique for them because they might be remote. We want to make some progress on the fundamental building blocks of a remote protocol. How can we make iterative progress on this topic? That when we come back in a year’s time there will have been

progress?

brrian, Being able to set breakpoints is something I actually added to Safari. I added a capability that would pre-load devtools so that a debugger statement would pause.

brrian, There are other things you could do, such as showing the commands coming in—for example in your devtools console.

brrian, I'm not sure setting breakpoints is the right way to do that, but it is useful.

brrian, WebDriver can launch the browser itself so you can not assume these things will be open.

<brrian> https://docs.oracle.com/javase/8/docs/platform/jpda/jdwp/jdwp-protocol.html

brrian, The Java debugging protocol is complicated.

brrian, It's hard to get debugging right.

brrian, [because there are unique constraints with browsers, how they load/interpret JS, &c.]

jgraham, I thought it sounded like there wasn't desire to define the initialisation sequence.

jgraham, The point was more that browsers have different, unique ways of starting up devtools.

jgraham, For the test use case I think it makes sense to have a standard way of handling the initialisation.

jgraham, There might still be browser specific steps to get the browser into the state, but once you get that in place, there should be a standardised way of getting the two-way protocol set up.

jgraham, That should not be any less standardised than setting up WebDriver.

jgraham, And to brrian's point, I think we should independently of this effort have a way to attach a debugger.

simonstewart, I wouldn't worry about how to start up the driver.

simonstewart, Leave that to local clients to figure out.

simonstewart, If you take a look at the WebDriver support libraries, they will first download geckodriver/chromedriver. Even Puppeteer also has this. This is a solved problem.

simonstewart, Upgrading the connection sounds like a good idea.

simonstewart, It would be desirable to expose this in a way that would let us loop back into the browser.

BrendynA, Are you saying the startup path would need to go through WebDriver?

simonstewart, No. I'm saying it would be possible to upgrade to it through WebDriver. HTTP/JSON/WebSockets.

simonstewart, That would be passed straight through to how the browsers implement this. We don't define how they implement the remote protocol.

simonstewart, How do we implement bi-direction communication? What primitive should we be looking at provoding through the WebDriver? And how do we make this simple to implement on top what existing browser already do.

BrendynA, It is bigger from our end than WebDriver.

BrendynA, Is this the home to drive these discussions going forward?

simonstewart, My personal opinion is that the Browser Tools and Testing WG is the perfect home to talk about browser tools and testing.

AutomatedTester, There is a charter discussion, which is why MikeSmith is hanging around today.

AutomatedTester, The charter is incredibly fluffy by design.

AutomatedTester, “You can drive a bus through it”

AutomatedTester, Historically why this is in WICG is that people said they didn't want it as part of the W3C.

AutomatedTester, When the first charter was written six years the idea was to put [developer tools] things like dragonfly and scopes, from opera's devtools, into this working group eventually.

AutomatedTester, The way we have structured the WebDriver specification is by calling out wherever possible to call into other specifications because it makes our lives easier.

AutomatedTester, Our core focus is using the transport layer but that doesn't stop devtools from growing out and doing interesting and wonderful things.

AutomatedTester, I want to structure this in a way that doesn't make people feel they are being hindered.

corevo, One of the great advantages with CDP is that I can bootstrap it into any running tab in an already running browser.

corevo, One of the disadvantages (but also in some cases advantages) with WebDriver is that it owns the browser process.

<boaz> scribe: boaz

AutomatedTester, im trying to think through the spec implications

simonstewart, its selenium ide or a dev tools package

AutomatedTester, I think most developers would think that if the extension is there it is there

AutomatedTester, do you have to start up chrome differently to do this

corevo, no, you ask for the permission, and then I can attach to the page, the same way a user does.

simonstewart, the extension asks for the permission

corevo, in a chrome extension you ask for this

corevo, and then you can attach to this

calebrouleau, you just get a port, right

corevo, no, I get an object that I can talk to and I send it javascript objects, and that sends the stuff into the internals.

<simonstewart> https://developer.chrome.com/extensions/permission_warnings

AutomatedTester, in gecko terms, it is doing what xpcomm does

<simonstewart> If you ask for the "debugger" permission, you get access to https://developer.chrome.com/extensions/debugger

corevo, and I do see some differences between what is available via websockets and what is available though the extensions

ato, lots of things have been said

ato, there is a lot of prior art in this area.

ato, one of the reaosns i think we need to be somewhat ambiguous about how to get at this protovcol, is preceisley to allow for these different ways, like corveo's extension

ato, as a parallel in gecko, the we we implemented marrionette was to make a tcp connection with the browser

when I implemented my first debugger protocol at opera in 2008, i did the opposite, where we implemented it by the url

ato, I think we need to support all three ways... diff browsers need different security requirements, and different devices have other characteristics that we probably have not thought of yet.

<calebrouleau> whoops

ato, mozilla is working on gecko view, targeted at web vr, but if you embed this thing inside of an androud app, you dont have a way to say it has started, and you need some way of exposing it.

ato, we need to have a standard set up to initialization, but it would be useful (espectially for mobile devices) to leave startup ambiguous

AutomatedTester, we are going to have a break now. it is clear that everyone is in agreement that this would be useful.

AutomatedTester, we may need to defer this to the wicg to get a transport that we could use in this WG.

AutomatedTester, then future versions of webdriver could use this by design.

ato, why does this need to be defered to the wicg?

AutomatedTester, they have that specification?

ato, the wicg group was started because there wasn't significant buy in from every vendor.

AutomatedTester, no it was because it was put into that group mostly because of patents

boaz: I thought it was because the devtools team was reticent to standardize

ato, sounds like we're having a charter discussion

calebrouleau, I wonder if we can spec out features that users need without having that protocol.

jgraham, yas

jgraham, irrespective of where the work has, I think this is a good time to have more detailed conversation on this api.

jgraham, I think there was a list of things to discuss from microsoft

BrendynA, vscode users are our customers, so that list is based on what they need

https://github.com/WICG/devtools-protocol/blob/master/meetings/agenda-tpac2018.md

BrendynA, it sounds like for this group this is more focused on webdriver, and addressing webdriver user things

jgraham, I think there are two things and they will meet in the middle, and that will be unsatisfying

topic over

BREAK

<JohnJansen> RRSAgent: generate logs

<JohnJansen> RRSAgent: make logs

<JohnJansen> RRSAgent: make logs public

<JohnJansen> RRSAgent: make minutes

<JohnJansen> RRSAgent: make logs public

<sahazel> simonstewart: Tools protocol is just beginning the path toward standardization. It may be useful to come up with a standard for bidirectional communication without getting too much into details of the protocol beyond that.

<AutomatedTester> scribe: sahazel

<scribe> scribe: sahazel

simonstewart: Tools protocol is just beginning the path toward standardization. It may be useful to come up with a standard for bidirectional communication without getting too much into details of the protocol beyond that.

jgraham: this group could provide a list of features to include in a standard layer which would translate into e.g. CDP

BrendynA: one use case is using dev tools within a webdriver session

simonstewart: This specification needs to account for intermediary nodes, not just direct connections to browsers. Making a tools protocol URL available to WebDriver could make intermediary services possible and enable using dev tools during WebDriver sessions

<AutomatedTester> RRSAgent: make minutes

jgraham: not sure what distinction you're trying to draw. Why specify a bidirectional protocol that isn't the one we intend to end up with? Don't necessarily need to map existing WebDriver commands into bidirectional protocol. Suggest we come up with use cases we need to support.
... we need to specify features rather than what specific json messages should look like

ato: event-based model seems distinct from CDP details
... session initialization is interesting to discuss

BrendynA: it would be good to have a standard way to initiate a bidirectional protocol session, regardless of browser

jgraham: would it make sense to have a command upgrade a webdriver session to bi-directional

<ato> ack

BrendynA: not sure we'd need to convert to bi-directional rather than just open a bi-directional session simultaneously

ato: not sure that would make sense

<ato> reillyg: ←

reillyg: I'm the editor for webusb and other device-orientied APIs chrome is implementing
... we've very interested in having these APIs be automatable over webdriver

<reillyg> https://wicg.github.io/webusb/test/

reillyg: we have an API that specifies how to automate devices, could be a good webdriver extension
... what we need is the ability to enumerate available devices, allow the test runner to choose a device
... need permissions for device access, bi-directional API would certainly help

calebrouleau: don't we have permissions specified for webdriver?

brrian: I added capabilities for requesting access to the camera (always a mock device)

<calebrouleau> is this not the API https://w3c.github.io/permissions/#automation?

reillyg: I have a request from developers who have actual devices connected to a machine, and would like the ability to request access to a device
... main concern I have is we don't really trust WebDriver sessions. At least in Safari, we use a clean session state and don't allow access to cookies, secrets, camera, etc due to security concerns

ato: looking at this webusb testing API, this seems to be some kind of privileged API?

reillyg: yes, trusted API, designed to plug in extensions for testing on devices
... using this as an example of the kind of interface that would be useful. changes would be needed to fit webdriver and be useful for developer use cases

ato: is the remote part primarily the interest?

reillyg: the point is automation. remove vs. local is not the concern. primary dev use case is making the permission decision automatable.

ato: before lunch we had a long discussion about standardizing something resembling CDP. CDP is structured so that it has multiple services. We added permissions API to webdriver, and we're now discussing a way to upgrade from webdriver to bi-directional. Might it make sense to put this in something like CDP, as a new service?

reillyg: not familiar with this group's thoughts about standardizing parts of CDP. seems orthogonal.

jgraham: I think permissions are relatively well scoped. Request makes a lot of sense. We had a spec. Is it worth doing permissions as part of the level 1 protocol, vs. making it something that has to use the bi-directional protocol?
... e.g. responding to a prompt could come back as an event over bi-directional

reillyg: we tried a less bi-directional way of doing this that had edge case problems, migrated toward using events, which works better

brrian: isn't it problematic to have something waiting in this way?

(some disagreement on how much of a problem that is)

ato: there's talk that using a ring buffer isn't the best model for logs, and an bi-directional, event-based model might be better suited. permissions may be similar.

jgraham: are permissions grant/reset, or responding to prompts?

<AutomatedTester> https://w3c.github.io/permissions/#automation

<boaz> /w3c.github.io/permissions/#automation///w3c.github.io/permissions/#automation

calebrouleau: unsure. permission editors draft has a simple set permission...

jgraham: unclear if there's broad agreement on that spec. it may be possible to implement automation parts without implementing the spec.
... in the permissions spec there's an endpoint that allows you to set a permission. I believe, could be wrong, mozilla has concerns about the spec constraining what the DOM API could look like. Would need to check with that group.

oops, yes, autocompleted wrong. "main concern I have is we don't really trust WebDriver sessions" comment was from brrian

JohnJansen: what's the topic?

AutomatedTester: detour into permissions spec

<JohnJansen> q

ato: it seems like there's a range of things that have requirements around permissions

jgraham: don't think it's controversial that permissions are needed in a testing API. Lots of examples of where they're used. Question is whether bi-directional is needed for permissions, and sounds like the answer is no.

<AutomatedTester> RRSAgent: make minutes

JohnJansen: I think what James just said is what should be the point of this conversation: we know there needs to be an automated way to address things like permissions for camera, permissions are needed for that, does that require 2-way sync? no? okay that's one use case, let's move on. Or else, we can end the 2-way topic, an move on to permissions. shouldn't have both conversations at once.

<ato> r?

reillyg: I do have a user scenario for permissions that requires 2-way. Event-driven "there's a prompt, how should I respond?" is a more reliable model.

<brrian> r-

soareschen: In the webrtc working group, we're trying to automated a test for getusermedia which has a permission prompt. We're not sure of the implementation status of the spec. Do people here know about the implementation plan?

jgraham: not sure of mozilla's status, but I have a feeling people are concerned it pre-supposes a specific UI

soareschen: bi-directional would allow simulated prompts so that tests can test the behavior of when a user permission is being requested

ato: I could see the concern about that being too tightly tied to browser UI. Mobile may differ from web.

calebrouleau: does that change whether we need bi-directional?

<ato> ack

ato: I thought there was agreement we should do permissions, current status is no one has implemented it.

calebrouleau: Chrome is looking into it an may implement it

brrian: the feedback we've gotten is people have been able to test what they wanted to test with a non-bidirectional model. but, they're only using one device. Don't know if multiple devices changes things.
... on mobile there's a lot of experimentation about how to determine whether a permission has been granted. Can't really simulate a generic permission dialog because those decisions can depend on anything. Shouldn't pretend doorhangers are normal.

AutomatedTester: more on use cases, or should we have an action to collect them?

jgraham: maybe Sauce wants to talk about their experience using WebDriver + CDP? what functionality makes it attractive?

Use cases for remote protocol services

ChristianBromann: we have some custom commands added in Sauce: throttling network, custom log types, including fetching network information
... we're now adding custom commands to fetch performance information
... we have a middleware between ChromeDriver and chrome and we start and stop tracing during commands
... we have a long list of things we could imagine adding

ato: you mentioned several different types of logging, that seems important

simonstewart: one of the reasons people want bi-directional is so they can spool logs out of the browser, esp. js errors, console logs, while tests are running

sahazel: people use this when tests break, to diagnose

ato: we had something like this for subscribing to logs. Works a lot better than capturing logs in a ring buffer.
... there are a lot of use cases. getting something like puppeteer working is most important for me

jgraham: I have a list of all the puppeteer commands, but not whether they're important or esoteric. logging is one of them.

<ChristianBromann> puppeteer command docs: https://pptr.dev/

ato: BrendynA is more interested in debugging, there's also discussion of usb

AutomatedTester: maybe time to dive into a something specific. Is there something we can scope, like logging, and then we can gather the rest asynchronously and use that structure?

ato: I wonder whether to make the list of use cases come together, if we should have someone around the table collect use cases?

AutomatedTester suggests selecting something concrete we can work on as a group

ChristianBromann: something we see as useful is the ability to capture data of various kinds, or act on certain events

AutomatedTester: suggest we describe how to handle logging

jgraham: we can talk about logging. but at what level is the conversation happening. CDP has a logging API. Is that sufficient?

ato: does the API work, what types of logging does it provide us with, and is it possible to standardize those types? network logging could be hard.

jgraham: looking at the devtools protocol viewer, the endpoint that existing at the moment is you can enable, clear, and disable logs, it sends you a bunch of events with a logentry object, which has a source string, a level string, free-form test, timestamp, optional URL, line number, and stack trace...

<jgraham> https://chromedevtools.github.io/devtools-protocol/tot/Log

calebrouleau: what are your use cases for logs (Sauce)?

Logging

ChristianBromann: we allow them to capture logs and give them all the logs from the time the test was started to the current moment
... there are a lot of events in different domains that could be a log type that would be interesting to users

AutomatedTester: I didn't appreciate how much they were doing. We could start here. Seems maybe slightly limited by type of info we want to get out. May need to revisit prior agreement on structure of log info.

<AutomatedTester> RRSAgent: make minutes

ato: I think we've come to the realization there are some types of logs we could agree on a standard for: console logs, network info. There are other things where it might be more difficult, but that doesn't mean browsers should be able to deal with those kinds of logs. There may be chrome- or firefox-specific log types. As long as the log domain API is well specced, it doesn't really matter what the browser returns.
... but it might make sense to have something in the spec about what console logs should look like

jgraham: there will be a subset everyone can agree on
... there's other stuff that is maybe more complex. Console logging seems fairly straightforward, unless there are other requirements...

<AutomatedTester> scribe: AutomatedTester

Loggin

<scribe> scribe: sahazel

ChristianBromann: network logs will be difficult because there's a lot of potential info to include

<AutomatedTester> RRSAgent: make minutes

<AutomatedTester> RRSAgent: Make logs public

ato: for example I wouldn't associate the ability to add cookies to network logs in firefox. Need to discuss which parts to standardize. I can see already we can't standardize all the network stuff.

<boaz> https://console.spec.whatwg.org/

brrian: just want to enumerate some things in webinspector that aren't specced
... what happens when you console log an object? not specced
... there's a very detailed media logging feature added last year

<boaz> looks like domenic was filiing issues on console this week

<boaz> nad folks working on this as recenlty as july: https://github.com/whatwg/console/commits/master

AutomatedTester: historically we were talking about different types of logs, and you could ask for different types of logs by name

brrian: we also do some limited network logging, though mostly we provide HAR files

AutomatedTester: biggest use case I keep getting at mozilla is from Angular and React teams, because they want performance info
... when React is seen as slow, people rarely think it's the browser, they think it's React. Web developers blame React first before other things, even though it may be their own js.
... not every browser describes performance in the same way, so how would that be structured?

boaz: like profiling, "how long did the browser spend on this function?"

ChristianBromann: what we did at Sauce is we have a middle man that starts tracing before page is loaded, stops tracing after. Internally we have tools that pull info out of the tracing info from Chrome.

boaz: the behavior I'm thinking of is there's something slow, dev profiles in the browser, tries to optimize code, wants to write some kind of test case to make sure it doesn't get slow again

ato: maybe it's fine for some of these log types not to be specified. Maybe performance logs can just be some browser-specific data

AutomatedTester: yeah, React team doesn't care if Firefox perf info matches Chrome

ato: second point, it would be interesting to return non-string types, but we are sort of constrained by CDP

BrendynA: one practical approach: there's a subset of log types that EDP and CDP have implemented. We could start with those. Go with what's implemented and build some WebDriver experience around that.

JohnJansen: are people happy with them?

BrendynA: think so; nice starting point because we have two browsers that implement this

<JohnJansen> +1 to BrendynA

<ato> BrendynA++

jgraham: it seems like when people say "logs" they mean a lot of different things. Seems fairly straightforward to see how to proceed with console logs as strings. Something we probably all want to implement in the same way, easy to see what to do. Once you throw in network, perf, it's unclear there's enough commonality to talk about in a standards setting.
... don't think "you should be able to enable it but every browser sends completely different data" is a good place to begin
... even though the more complicated stuff seems important, it's going to take longer to bake

brrian: that sort of thing is better for the actual dev tools protocol, not really an automation thing

ChristianBromann: could split things based on whether the user will be interested in the data directly, vs. providing data to dev tools

ato: don't think this is the right group to define what goes into a performance log
... it's a use case we should record that we want to have perf log, we're the right group to make strides on putting in place the plumbing, but perhaps perf log discussion needs to happen in a group that's more suited, once we have plumbing in place

boaz: I think what will be important in perf reporting will change. two constituencies: browser developers who understand what affects perf, and web developers who use perf tools

<brrian> carry on. fetching water

<ato> Scribe: ato

Group Charter

MikeSmith, We are actually not chartered anymore. This is not currently a problem since work is still happening.

MikeSmith, But it is a problem if when we need to publish again.

MikeSmith, Patent commitments also requires being chartered.

MikeSmith, We have to have a charter that reflects the agreement from the group about what we should be working on. There should only be things in the charter that is within the scope of the group.

MikeSmith, We have the same charter as what we had before. The one we've been using all along.

<AutomatedTester> https://w3c.github.io/charter-drafts/browser-testing-tools.html

MikeSmith, That’s it.

MikeSmith, If you look at that there have been some minor changes.

MikeSmith, Originally it said in the scope section that it included “API for automated testing”.

MikeSmith, A lot of people said this doesn’t look like a web API and I’ve changed it to say "protocol" instead.

MikeSmith, Otherwise most of it is boilerplate, except the normative [?], a single item: WebDriver.

MikeSmith, In 2.1 we've added "the group may publish other normative specification within the scope define in this charter".

MikeSmith, This is to stop people from asking us why we're doing other work than just WebDriver in the future.

MikeSmith, Otherwise the rest of this is uninteresting.

MikeSmith, Questions? Does anyone takes issue with the language? We can refine it and make further changes.

MikeSmith, plh and I plan to take this to W3C management, and then to the AC. These are process steps, but they take time. 3-4 weeks to review. Even if we start this next week we're realistically looking at 2-3 months before we are chartered again.

AutomatedTester, Missing Apple Safari and IE.

MikeSmith, Yeah.

brrian, Engines or browsers?

jgraham, It says "engines".

MikeSmith: Remove Servo.

[discussion about difference between Chromium and Blink]

<AutomatedTester> scribe: automatedtester

ato: Is there a reason why this is not a CG and why would we want to publish

jgraham: We had this discussion last year. the WG has some patent protections
... 1 option is we follow the W3C process
... or the other option is to do the WHATWG process
... having had a working group seems like the path of the least of resistance

ato: We discussed in the past about moving to REC, we saw a lot of process. Collecting the data results was hard to get

<JohnJansen> RRSAgent make minutes

<JohnJansen> RRSAgent: make minutes

AutomatedTester: that is a lot easier now with WPT.fyi. we can fix any issues where it is not an issue that we can do

<JohnJansen> last year: https://www.w3.org/2017/11/09-webdriver-minutes.html#item01

ato: what concrete steps can we take to make sure that we don't have this discussion each year
... we have adopted the change where we need tests for each PR.

MikeSmith: The process is where we need tests and implementations. The WHATWG is the same as the W3C. This group has fixed the process
... with the rest of the process it is what it is and things do take time
... as for publishing every 6 months the WASM group is already doing that
... The WASM group focus on specific features and then get that to being published
... if we had to do a snap shot every 6 months we would need to know what we were going to focus on

<ato> RRSAgent: make minutes

jgraham: we did talk about this last year. We talked about having a way to pref features off

MikeSmith: we had this discussion last year and it feels like we want to keep it going.

ato: What are we working towards, what is the next version going to be called?

MikeSmith: I don't know, this is a bad side to needing to be REC. We need to be able to link to different versions. The simplest way is to have it with a number and just increment it
... We should just make sure that we are going to supersede the previous group and show that it is now obsolete

jgraham: we can call it SNAPSHOT-2019

MikeSmith: the process doesn't suggest what it should be called

ato: I quite like the idea of calling it "SNAPSHOT-*"

MikeSmith: we just need to make sure we keep directing people to the ED

ato: Is there any reason with W3C Process to have Living Document isn't a thing?

MikeSmith: I think its just a tooling issues

ato: MDN was linking to the TR and I wanted it to be linking to the Living Standard
... MDN said they can't be linking to ED

<scribe> ACTION: Can we have something to tell people not to read the TR

RRSAgent: make minutes

<JohnJansen> RRSAgent: make logs public

ato: How often would we need to do a release candence.

MikeSmith: This is what is decided on the group.

ato: what if we just maintained the living document

MikeSmith: we need to release something so that we can get the patent protections

ato: it is up to the editors how often they want the pain

BrendynA: I want a clarification about "The group may publish other normative specifications within the scope defined in this charter. "

MikeSmith: this allows us to do as much as we want and we could bring in the dev tools protocol.

JohnJansen: we thought the webdriver spec would have been quicker

all: yup

RRSAgent: make minutes

<whimboo> RRSAgent: make logs public

Upgrade path from webdriver to bidirectional webdriver

jgraham: we have agreed on using a bidi comms tool and we should decide on a upgrade path
... simonstewart had a design and I have a design
... I had been thinking about this about webdriver having a different end point
... there are pros and cons to each approach
... pro is that you can do everything you want and if you hit this end point you can then grow the feature
... con there is extra complexity around the semantics

ChristianBromann: I see the immediate necessity from having that.
... I can see doing more parallel requests

jgraham: so with the session response

ChristianBromann: yes. with webdriver.io I would automatically upgrade
... with chromedriver you get the details in the capabilities

jgraham: I see that but with Sauce you wouldnt want to have to open up this connection all the time because that would be a lot of resources

brrian: what is the scope of this communications? Is it per tab etc?

BrendynA: it could anything, it's a target like a webview/tab

brrian: would we need to describe this

ato: so is there one connection for everything?

BrendynA: no, there is 1 WS url for each target which is a tab/webview

ato: so you cant start it up all

jgraham: we dont need to follow CDP, we could have an endpoint that multiplex to other end points

simonstewart: there could be a use case where people have multiple tabs and then try drive all of them

jgraham: the comms for bidi webdriver would have an id
... and then just make sure it routes correctly

BrendynA: CDP emits an event when there is a new target
... and you can subscribe to that item

ato: what would the API look like then?

simonstewart: there would be 2 target domains

jgraham: the exact details on how CDP works is not for this group, we can hide over this.

BrendynA: you need an initial WS to gives you the other info on the other WS endpoints

simonstewart: the way that CDP works at the moment, you start up a server and hit a HTTP server and that gives you info those details. We can do New Session and then in the returned caps it would have details of where to look for the new bidi world

jgraham: in both models it's where do we want the scope to be tied to.

brrian: for webkit we have daemon that speaks to all the apps that are debuggable
... we are moving from connecting from connecting to the content to the actual UI app.
... with webkit we are also going to be able to connect to things like webview and service workers

<JohnJansen> RRSAgent: Make minutes

<JohnJansen> RRSAgent: Make Logs public

BrendynA: will that block the ability for 2 apps trying to debug. E.g. VS CODE and Safari devtools

brrian: I do worry about all the traffic from iframe heavy pages and having to build up connections for each

sahazel: I am little unclear if this a one time upgrade. Would the WS and the http connection be running in parallel

jgraham: this seems like its running along side

simonstewart: if you expose then people can decide over time. It is easier to do a sync API over async APIs

jgraham: not all commands would make sense to have a CDP domain.
... there are use cases were people don't want a 1:1 mapping of webdriver to puppeteer tools

calebrouleau: I am confused to where this would all live. Would this would be ?
....: This would be on chromedriver

BrendynA: why would you do this in chromedriver vs doing directly to the browser

jgraham: the comms might be slightly different and then chromedriver would make sure that it would speak directly to chrome
... and you would need it for the being the multiplexer
... I can see good reasons why were would move this to chrome
... you would need something that starts chrome in the right way, either the client bindings or chrome driver
... if saucelabs grabs that URL and they could augment details

<calebrouleau> ?

BrendynA: what is the concern with 1 WS per target vs 1 WS per session

jgraham: it is an easier model

simonstewart: CDP is moving to a multiplexer
... it also makes it easier for the likes of Saucelabs

AutomatedTester: Windows also suffers from low ephemeral ports

brrian: I dont really care where the WS lives but here is why it wont be in the browser. What if an app has a webview and we have open a random port, the developers would be really annoyed
... on a phone it would be really hard for a WS to be running, it would use a lot of resources
... we have a system daemon that routes this info for us
... we would probaby host this via safaridriver

ato: what is the approach we have agreed on

jgraham: have a capability with a URL vs a capability with a boolean and a different end point
... we would need to have a capability to opt in on new session

ato: what happens if you don't return the capability?

Zakim: excuse us

RESOLUTION: There will be a capability that lets you ask for a CDP-derived remote protocol. This will take a boolean, and if true and the remote end can fulfil the capability request, will return a string URL to the WebSocket for said protocol. It will return null by default.

RESOLUTION: The WebSocket endpoint will be hosted by the WebDriver remote end and act as a multiplexer for CDP target domains (browsing contexts, servicework, et al.). This will take a web window ID to identify the target domain.

RRSAgent: make minutes

- DRAFT -

Browser Tools and Testing WG, day 1

25 Oct 2018

Attendees

Contents

Setting the Agenda

Documenting WebDriver on MDN

Chrome Devtools Protocol

Use cases for remote protocol services

Logging

Loggin

Group Charter

Upgrade path from webdriver to bidirectional webdriver

Summary of Action Items

Summary of Resolutions

Scribe.perl diagnostic output