<Hexcles> ScribeNick: boazsender

foolip: the goal of the session is to gather wider input

flackr: im working on web animations, wpt.fyi is running all of the release browsers with no flags, and we do our development with flags, and it would nice to have test results for in development features

jgraham: the setup at the moment, you can use prefs locally

<foolip> https://github.com/w3c/wptdashboard/issues/146 is the issue about getting both stable and nightly browsers

<foolip> "Have two groups of browsers: stable and latest with experimental features enabled" is the title

foolip: we want two groups of each browser
... both stable, and in development with flags

flackr: im fine with stable being the default
... that makes sense if the target is developers

dominic: who is the target

foolip: implementers
... there is a weird mix of stable, experimental flags. we want to imrove this.
... the challenge is to get edge insider and safari tp, we are working on this

<xfq> Meeting: web-platform-tests

johnjansen: the other side of this says that you want to make sure your changes dont regress a browser without the flags enabled
... locally we run with the flags enabled, and expose them publically

<zcorpan> (Y'all can put your names in https://docs.google.com/spreadsheets/d/1zgrl56n7KKh5WvsjGU4oH0Ft6vqK-m0P9mG3bURaK1s/edit?usp=sharing to help the scribe)

dominic: its not easy to run the test locally, esp on windows

johnjansen: i edited the readme for windows. it works really well for the bash shell on windows.

running locally

dominic: node ecosystem has a similar issue with windows and deps. they created a one click installer. idk if its worth spending time on, but it would be great.

jgraham, I think that the only deps you need is python and virtual env. you dont need openssl anymore.

Domenic: what about for the ssl certs?

gsnedders: you do not, we have a pregenerated cert.

foolip: we could have a windows ci environment to make this better.

jgraham: appveyor exists, but its too old.

gsnedders: historically most of our issues have been with path separator bugs.

johnjansen: we talked on public test infra about getting windows infrastructure on browser stack

foolip: we are going to switch t obrowser stack
... what would be amazing would be for wpt run to be able to send to browser stack

<scribe> scribenick: potch

johnjansen: I'm personally not familiar with executing in that regard

<gitbot> [13web-platform-tests] 15gsnedders closed pull request #8110: Sync Mozilla reftests as of 2017-11-08 (06master...06mozilla-reftest-export-20171108) 02https://github.com/w3c/web-platform-tests/pull/8110

boazsender: the topic is making it easier to run for the first time
... at bocoup, looking to run on local infrastructure. copy of windows insider setup. 3 versions of insider. Might be possible to improve experience

foolip: as far as resourcing, if the edge team does this it would be convenient

<gitbot> [13web-platform-tests] 15jgraham tagged 06merge_pr_8110 at 06master: 02https://github.com/w3c/web-platform-tests/commits/merge_pr_8110

john: we have an internal script to setup (hacky) to setup environment

boazsender: would you send that to list when it's public

<inserted> john: yeah

foolip: did you give up on win testing?

Domenic: found a python command that worked

foolip: pause for other topics

<breezet> here

<breezet> Qingqian Tao from baidu

breezet: question: do we have a solution to mobile web/browser testing?

mobile testing

boazsender: we don't currently have a workflow; as part of explanation of testing locally, next step is to collect from other devices

breezet: many chinese mobile browsers

<foolip> I have filed https://github.com/w3c/wptdashboard/issues/219

boazsender: want to cover those for sure. we want to be collecting that data- headlessly or other tools in the node community for multi-touch robotic driving

breezet: do we have a solution to test mobile browsers

jgraham: depends on what you mean. what we don't have is mobile browsers to test in. we havne't set up the infrastructure. we didn't design infra that doesn't work with mobile, we haven't done the work

<Hexcles> Robert Ma from Google

gsnedders: with chinese browsers, no webdriver support

<kereliuk> me

even if we had emulators, we still need webdriver from those vendors

<kereliuk> me

kereliuk: I work on chromedriver, talking to Hexcles before, many chinese mobile browseres chromium forks. could run on modified webdriver.

foolip: if UC browser thiinks wpt is valuable, we should get them on the dashboard if we can get it to work

trying mobilesafari already, makes sense to try first

<Guest73> here

<foolip> gsnedders: sssshhh

rniwa: we are running WPT in simulator, we don't have to use webdriver there, we have our own harness. discussing building harness api. It's a lot of work for a vendor. Most browsers have their own testing API. One workaround is to provide an overwritable/extensible interface to run the tests

especially due to mobile, it's very hard to support entirety of webdriver

<nigel> Meeting: TPAC 2017 Testing breakout

jgraham: webdriver is not a requirement. for the dashboard right now. existing impl uses webdriver for many esxiting browsers. but servo uses another method. No actual webdriver req, except for testdriver, but you can provide your own implementation. This was designed into the wpt runner in the repository

for specific vendors, they may want their own setup. but for the dashboard, we can support other methods. EG TV browser, you can only use the directional remote

foolip: this may not be an issue. getting UC browser on the dash may jsut wor

work*

<gitbot> [13web-platform-tests] 15chromium-wpt-export-bot 04force-pushed 06chromium-export-cl-753262 from 14d354cb3 to 144376b21: 02https://github.com/w3c/web-platform-tests/commits/chromium-export-cl-753262

<gitbot> 13web-platform-tests/06chromium-export-cl-753262 144376b21 15Matt Falkenhagen: service worker: Upstream websocket test....

johnjansen: a note about the dash, may be useful to separate mobile.

foolip: may be good to represent the engines (browsers are proxy for this), but then to list actual browsers

<inserted> Testing WebBluetooth, WebUSB, WebVR, etc.

Topic: Testing WebBluetooth, WebUSB, WebVR, etc.

<scheib> scheib here

<gitbot> [13web-platform-tests] 15jgraham created 06cache_testharness (+1 new commit): 02https://github.com/w3c/web-platform-tests/commit/9b4d95e1a214

<gitbot> 13web-platform-tests/06cache_testharness 149b4d95e 15James Graham: Add cache-control headers for testharness* files....

<gitbot> [13web-platform-tests] 15jgraham opened pull request #8115: Add cache-control headers for testharness* files. (06master...06cache_testharness) 02https://github.com/w3c/web-platform-tests/pull/8115

scheib: testdriver mentioned as can run on top of webdriver, but other impls as necessary. WE just saw trusted click, the ability to simulate user click, required for things like fullscreen. convo on webusb, bluetooth, vr, do people have thoughts on if a feature needs a new API surface. E.G. need to simulate a bluetooth device. Should we keep that bundled in a helper lib? Or add capabilities to testdriver directly. Rather not reinvent the wheel to

expose features that need impl specific backends

clmartin: we discussed yesterday; we should think if it makes sense to add to spec, we want to make testing better. if not viable in short term, testdriver.js good for that

ato: to clarify, there is a 1pm breakout session on VR and testing. similar need for testing. don't know about webvr. they have a simulation library that lets you smooth over impl differences. Their need is special, but still an open question on how to extend webdriver spec. There is language for how to add specific additions for testing. That may be an option. we haven't had specs attempt yet, but there are primitives in webdriver. Others should

speak to how to model the libraries that extend. My gut feeling is if webdriver is for testing the web platform, best not to complicate.

jgraham: don't think the situation has changed substantially. if we talk about fake bluetooth devices, there may be use. If your spec specs a testing interface, we will try to use those. I don't think that tesdriver is involved. point is to abtract over browser diffrences between implementations.

foolip: I agree with value of that, there are cases where it's not clear where things should change. maybe it's testdriver, maybe wd spec will change.

<gitbot> [13web-platform-tests] 15chromium-wpt-export-bot 04force-pushed 06chromium-export-cl-710516 from 147cb39bb to 148b47775: 02https://github.com/w3c/web-platform-tests/commits/chromium-export-cl-710516

<gitbot> 13web-platform-tests/06chromium-export-cl-710516 148b47775 15Kim Paulhamus: Hook WebAuthN to //device/u2f to perform credential registration....

boazsender: another pov, how to synthesize test API in spec. May be other consumers like devtools

<gitbot> [13web-platform-tests] 15chromium-wpt-export-bot closed pull request #8065: service worker: Upstream websocket test. (06master...06chromium-export-cl-753262) 02https://github.com/w3c/web-platform-tests/pull/8065

<gitbot> [13web-platform-tests] 15chromium-wpt-export-bot 04deleted 06chromium-export-cl-753262 at 144376b21: 02https://github.com/w3c/web-platform-tests/commit/4376b21

no consensus on where they should be specified, separate from practicality of impl, impl now and adjust later

clmartin: for media, allowences in spec for vendor prefixed commands. we've added some to unblock testing. for new spaces, immediate benefit if not permanent

rniwa: useful for webdrivers to consistenly trigger things like copy paste across platforms, even when env is remapped. Would be good to have an abstraction for these things instead of having to write wd code for each browser

foolip: lots of things we could add there

ato: I think there are limitations due to how copy paste works for example, not really a primitive in the spec for selecting, what we need is a text selection primitive. discussed in the past. could define this. to the point earlier about vr/bluetooth testing. I don't think the hard part is instrumenting for control, but harder ? is how to simulate the actual hardware/device under test. I don't know that yet, given limitations of webdriver API.

<boazsender> here is the usb testing API: https://wicg.github.io/webusb/test/

scheib: several points raised, it's possible to get anything into any shape theoretically. possible to get things into WD even in current shape, but practical details mean things can be very large, like 100-200kb. when usb landed test, the testing aPI is provided via js and a command line option that provides a mock interface. good, but very impl specific. To do this, we came up with our own abstraction, and chromiums specific detection and behavior. now

with testdriver, we could perhaps a single pattern instead of reinventing each time. Bluetooth has gone through refactoring, do we use the WebUSB design, webdriver design? looking for more specific okay to pile on to testdriver.

foolip: to summarize, we should be open to having testdriver be more things if that gets us more tests, always tricky. Let's open up for other questions

<gitbot> [13web-platform-tests] 15chromium-wpt-export-bot 04force-pushed 06chromium-export-cl-759023 from 147faf4f4 to 14c619d7f: 02https://github.com/w3c/web-platform-tests/commits/chromium-export-cl-759023

<gitbot> 13web-platform-tests/06chromium-export-cl-759023 14c619d7f 15Raymond Toy: Move WPT chrome constant source node tests to final place...

<rniwa> +q

<gitbot> [13web-platform-tests] 15jgraham tagged 06merge_pr_8065 at 06master: 02https://github.com/w3c/web-platform-tests/commits/merge_pr_8065

zcorpan_: I think it's exciting, can solve lots of manual test problems, but it will be difficult to reach 100% automatability. useful to think about what's practical to automate, maybe using robots to interact with screen. An actual robot that touches the screen, do multitouch, interact with permission dialogues, pretend to be a human to do things humans can do without modifying implementation. some things difficult for folks like safari. A robot

is cheaper than humans fwiw. Robot may be fast enough for 10 times a day. For some tests, internal automation might not get there. Wanted to ask for feedback

<wilhelm> https://www.tapster.io/

anssik: we like this, so we can flip devices for example

ato: in past, I dabbled with Firefox OS when that was a thing, I was asked to make a test harness for a physical device. Advanced robot testing is an actual thing for OEMs etc. When working on FirefoxOS, basically used webdriver to run assertions but with human control. The computer would instruct manual testers, and use webdriver to validate assertion.

foolip: I work in the office with webrtc, they use robot eyes (Cameras) to test connections. You can simulate the user!

rniwa: can we queue on IRC?
... another feature we have trouble testing is drag and drop; on iOS this is not automatable at all unless you automate physically. to some extent even the browser can't automate due to platform. Robot may be far fetched, but may be required.

jrossi: we have such a device, we use it for touch testing/perf validation
... happy to take an action to see if this still exists

<scheib> reillyeon:

reillyeon: this question of things like interaction testing and physical interaction. useful in product testing. but for platform tests, if these things can be mocked out in tests in the name of testing, this is appropriate for web platform tests. tests define behavior limited to scope, the browser and platform is outside the test.
... I'm not sure that physical tests are what we want to be doing.
... I can add, for permissions, we have permissions API to help standardize this, may be useful to have a companion API for testing user confirmation without needing UI mocking.

ato: permissions API is something we are going to add to webdriver. Not at the level of doorhangers and widgets, but closer to 'confirm/deny'. Not fully formed yet.

foolip: problem that webdriver working group not fully aware.

clmartin: I would add, when going through this for webdriver, for customers (browsers etc), give consideration to browsers putting things in webdriver, we want it to be broadly useful

boazsender: the other approach to permissions, is to introduce automating them in the spec, along the philosophical lines I mentioned earlier

about to drop a link

<boazsender> here is another approach for permissions automation: https://github.com/w3c/permissions/issues/153

This is scribe.perl Revision: 1.152  of Date: 2017/02/06 11:04:15  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Succeeded: s/I'm personally not familiar/johnjansen: I'm personally not familiar/
Succeeded: i/foolip: did you give up on win testing/john: yeah
Succeeded: s/a note about the dash/johnjansen: a note about the dash/
Succeeded: s/several points raised/scheib: several points raised/
Succeeded: i/scheib here/Testing WebBluetooth, WebUSB, WebVR, etc.
Succeeded: i/scheib here/Topic: Testing WebBluetooth, WebUSB, WebVR, etc.
Succeeded: s/Testing WebBluetooth, WebUSB, WebVR, etc./Topic: Testing WebBluetooth, WebUSB, WebVR, etc./
Succeeded: s/dominic/Domenic/
Present: Rob_Flack Google Simon_Pieters Bocoup
Found ScribeNick: boazsender
Found ScribeNick: potch
Inferring Scribes: boazsender, potch
Scribes: boazsender, potch
ScribeNicks: boazsender, potch
Agenda: https://www.w3.org/wiki/TPAC/2017/Testing

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)