IRC log of testing on 2013-06-14

Timestamps are in UTC.

13:06:31 [RRSAgent]
RRSAgent has joined #testing
13:06:31 [RRSAgent]
logging to http://www.w3.org/2013/06/14-testing-irc
13:07:42 [wilhelm]
Meeting: WebDriver F2F, June 14th 2013
13:08:46 [fisherii]
fisherii has joined #testing
13:09:08 [AutomatedTester]
AutomatedTester has joined #testing
13:10:18 [wilhelm]
Agenda: http://www.w3.org/wiki/WebDriver/2013-June-F2F
13:11:25 [wilhelm]
Chair: sstewart6
13:11:29 [wilhelm]
Scribe: wilhelm
13:11:39 [wilhelm]
RRSAgent, draft minutes
13:11:39 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html wilhelm
13:11:43 [wilhelm]
RRSAgent, make logs public
13:12:46 [wilhelm]
Topic: Test suite
13:12:54 [wilhelm]
Scribe: plh
13:13:14 [JohnJansen]
JohnJansen has joined #testing
13:13:31 [plh]
David: it was more working on it and writing tests
13:13:46 [AutomatedTester]
Present+ DavidBurns
13:13:48 [plh]
John: not sure if this is the venue or email but I'd like to understand the end-to-end plan for it
13:13:53 [plh]
Present+ plh
13:14:04 [plh]
David: it's in mercurial
13:14:05 [sstewart6]
https://dvcs.w3.org/hg/webdriver-test
13:14:16 [wilhelm]
Present+ Wilhelm
13:14:29 [plh]
David: we're trying to port the open source project tests
13:14:39 [plh]
... those have grown quite organically over the years
13:14:44 [plh]
... sort them, etc.
13:14:56 [plh]
... we're trying to move them across
13:15:10 [plh]
... there are parts when the tests and the spec diverge
13:15:14 [plh]
... like at TPAC
13:15:29 [plh]
... but we think the spec is correct and need to fix the tests
13:15:39 [plh]
Simon: we're taking the tests andmaking sure they map to the spec
13:16:12 [plh]
... once we're complete with the move, we can remove the tests in selenium
13:16:43 [plh]
plh: license for the tests?
13:16:50 [plh]
Simon: all Apache 2 tests
13:16:55 [plh]
... we're clean on that front
13:18:12 [plh]
John: is there a structure for submission?
13:18:18 [plh]
... like in the CSS WG
13:18:55 [plh]
Simon: correct, it would be better to have a holding
13:19:04 [sstewart6]
sstewart6 has joined #testing
13:21:50 [plh]
plh: [trying to motivate the group to move to web-platform-tests github]
13:23:08 [gdennis1]
gdennis1 has joined #testing
13:23:59 [kkania]
kkania has joined #testing
13:24:20 [AutomatedTester]
https://github.com/w3c/web-platform-tests
13:24:56 [chrisgao]
chrisgao has joined #testing
13:28:21 [plh]
All: looks like a compelling case
13:28:28 [plh]
Simon: let's do it then
13:29:34 [plh]
Resolution: webdriver tests will merge into web-platform-tests
13:30:30 [fisherii]
Present+ MarcFisher
13:30:37 [sstewart6]
Present+ SimonStewart
13:30:39 [kkania]
Present+ KenKania
13:30:52 [JohnJansen]
Present+ JohnJansen
13:30:54 [gdennis]
Present + GregDennis
13:31:08 [chrisgao]
Present + ChrisGao
13:31:21 [plh]
rrsagent, generate minutes
13:31:21 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html plh
13:31:54 [plh]
Simon: we already covered language in the test suite (python)
13:32:10 [plh]
Wilhelm: we should distribute chapters
13:32:19 [jgraham]
(not been following, but it's not clear to me that webdriver tests are much like the other tests there, or if they have the same license)
13:32:29 [plh]
Simon: will look 9, 10, 11
13:32:34 [plh]
... reading element state and executing JS
13:32:38 [jgraham]
(although there are ofc adcvantages e.g. you get free documentation)
13:32:44 [plh]
Wilhelm: I'll do 16 screenshots
13:33:13 [plh]
David: I'll do 10 and 17
13:33:18 [plh]
Simon: I'll do 9 and 11
13:33:59 [plh]
David: I'll do 14 as well
13:34:28 [sstewart6]
https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html
13:34:38 [plh]
(confusing about the spec version :)
13:34:43 [plh]
s/sing/sion/
13:34:51 [plh]
Ken: I'll do 5 and 6
13:34:58 [plh]
John: I'll do 2 and 3
13:36:18 [plh]
Chris: I'll do 15
13:36:41 [plh]
ACTION: Simon to figure testing for chapters 9 and 11
13:37:01 [plh]
ACTION: David to figure testing for chapters 10, 14 and 17
13:37:11 [plh]
ACTION: Chris to figure testing for chapters 2 and 3
13:37:20 [plh]
ACTION: Wilhelm to figure testing for chapters 16
13:37:36 [plh]
ACTION: Chris to figure testing for chapters 15
13:38:02 [plh]
ACTION: Ken to figure testing for chapters 5 and 6
13:38:26 [plh]
ACTION: John to figure out testing for chapter 1
13:38:55 [plh]
s/Chris to figure testing for chapters 2 and/John to figure testing for chapters 2 and/
13:39:02 [plh]
rrsagent, generate minutes
13:39:02 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html plh
13:39:38 [plh]
ACTIOn: Simon to create the web-platform-tests directory and move tests there
13:40:32 [plh]
Simon: we'll figure out the other chapters later
13:42:38 [plh]
[figuring out the next agenda item]
13:43:35 [plh]
Topic: SysApps
13:43:41 [sstewart6]
http://www.w3.org/wiki/System_Applications
13:43:51 [plh]
David: those are related to OS APIs. we're doing testing with marionette
13:44:40 [plh]
... we have JS libraries for access to things, like contacts, set geolocation, etc.
13:45:20 [plh]
... we do tests for batteries, bluetooth, livemaps, etc.
13:46:02 [plh]
... not sure I don't think we need to worry about it. we didn't extend marionette for them, we use the execute script method
13:46:17 [plh]
[Simon goes through the list of APIs]
13:46:30 [plh]
David: in Firefox OS we have a browser within a browser
13:46:34 [plh]
Simon: we must go deeper :)
13:46:44 [plh]
David: everything is based in an iframe
13:46:58 [plh]
... we try to prevent cross browser contamination
13:47:18 [plh]
... chrome os is doing similar things
13:47:44 [plh]
... a lot of these have come pout of Mozilla while working on firefox os
13:47:57 [sstewart6]
Specs appear to be here: https://github.com/sysapps
13:47:57 [plh]
... they are certain OEM who see value in it, like Samsung
13:48:14 [plh]
... don't think there are a lot of difference
13:48:29 [plh]
... the webdriver is low enough and don't need tweaking
13:48:59 [plh]
David: Marionette is the only I think I can use to test firefox os
13:49:21 [plh]
ACTION: David to double check there is no need to extend WebDriver because of Firefox OS
13:49:45 [plh]
Simon: it works because the OS is written in JS...
13:49:52 [plh]
David: agree but others would be out of scope
13:49:54 [plh]
Simon: ok
13:51:07 [plh]
Topic: Security dialogs
13:51:25 [sstewart6]
driver.authtenticateAs(Credentials)
13:52:15 [sstewart6]
http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/Alert.html#authenticateUsing(org.openqa.selenium.security.Credentials)
13:52:30 [plh]
Simon: it is currently unimplemented
13:52:41 [plh]
... it's there only as a local end API at the moment
13:53:07 [plh]
... this isn't handling things like OpenID
13:53:20 [plh]
... you would just write that use normal primitive
13:53:30 [plh]
... we model all modal dialogs as an alter
13:53:42 [plh]
s/alter/alert/
13:54:05 [plh]
John: this is targetting the security dialog but we didn't target modal dialogs yet, right?
13:54:12 [plh]
Simon: it's in the wired protocol
13:54:18 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/alert_text
13:54:28 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/accept_alert
13:54:34 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/dismiss_alert
13:54:47 [plh]
David: this is the open source wire protocol, not the google one
13:55:02 [plh]
Simon: this will be migrated into the missing appendix
13:55:39 [plh]
Simon: the way we handle dialog: we assume they all have the same level functionality: dismiss, cancel, send keys, get the text of the alter itself
13:55:59 [plh]
... make sure to fill a field, ie finding out what the prompt is about
13:56:18 [plh]
... because you're doing auth, you may do user/passwrd but there is a level of indirection
13:56:38 [plh]
Simon: do we want to allow webdriver to access basic digest? or is it out of scope?
13:56:54 [plh]
Wilhelm: right now, I'm sending keys to the browser
13:57:40 [plh]
John: automating a security dialog is a threat imho. it needs to be detailed and complete.
13:58:00 [plh]
... doing something that sounds insecure wouldn't fly in the company
13:58:21 [plh]
... we might not do it even if the spec says we have to
13:59:11 [plh]
... the limitation here that wouldn't be able to use webdriver for auth
13:59:26 [plh]
Wilhem: how can I auth my staging area?
13:59:31 [plh]
John: would need to auth first
13:59:39 [plh]
Soimon: but we can't connect to a running server
14:00:14 [plh]
Simon: on services like sourcelabs, you don't even have access...
14:00:20 [plh]
s/Soimon/Simon/
14:00:45 [sstewart6]
s/sourcelabs/Sauce Labs/
14:01:11 [plh]
David: for marionette, it's in the nightly. we got it approved: you would need to pass a flag
14:01:40 [plh]
... for desktop, it's command line flag. this goes through a number of checks, including certain preferences
14:01:50 [plh]
... if one is missing, marionette doesn't start
14:02:05 [plh]
... ie you have to do several steps to put you in an insecure state
14:02:21 [plh]
... for mobile, it's going to be different
14:02:35 [plh]
... we'll visit that once we're ready for the general public
14:02:54 [plh]
... that's how we got through the security review
14:03:10 [plh]
Simon: the command line turns the browser into something that is listening
14:03:28 [plh]
David: you need to have a number of commands in order to activate it
14:03:39 [plh]
... you can't remote over a network
14:03:55 [plh]
... you would need to have an intermediairy on the remote machine
14:04:19 [plh]
... you can't speak to marionette over the network
14:04:33 [plh]
... could be overriden but require deep knowledge of preferences
14:05:02 [plh]
... of course, if you can do silly things
14:05:28 [plh]
John: interesting approach even if 99.9% guarantee isn't enough
14:05:52 [plh]
John: in order to be hacked you need to make 3 or 4 wrong decisions
14:05:54 [plh]
David: correct
14:06:10 [plh]
Ken: it's a similar approach in Chrome
14:06:34 [plh]
... for android, not sure how the options get passed
14:06:54 [plh]
... since we don't speak wire protocol in chrome, you need a separate binary
14:07:07 [plh]
David: yes, we have a shime for the http protocol as well
14:07:20 [sstewart6]
s/ shime / shim /
14:08:17 [plh]
David: preventing add-ons from switching on marionette was vital
14:08:51 [plh]
... those aren't allowed in the store. those get automatically flagged
14:09:32 [plh]
... if you don't go through the store, all bets are off
14:10:42 [plh]
... I can find the security discussion meeting...
14:11:35 [plh]
Simon: if you've got an API for automated browser, spammer can use it. there are JS variables to detect webdriver was enabled.
14:11:47 [plh]
... should we agree on a fingerprint
14:11:53 [plh]
John: seems like a must to me
14:12:08 [plh]
... otherwise webdriver will be used against captcha
14:12:55 [AutomatedTester]
https://bugzilla.mozilla.org/show_bug.cgi?id=870576 https://wiki.mozilla.org/Security/Reviews/MarionetteCLIAll
14:13:30 [plh]
Simon: so, is our spec the right place to put that in and should it be the same in all browsers?
14:13:48 [plh]
Simon: initially, I was doing security through obscurity...
14:14:04 [plh]
John: the obscurity will get clear very quickly...
14:14:13 [plh]
... there has to be something else
14:14:38 [plh]
Simon: if you set an attribute on a document, it can get overriden...
14:14:52 [plh]
... in the ideal world, you would set a header
14:15:11 [plh]
... and you would set an attribute that cannot overriden, like document.automated
14:15:22 [plh]
... but then you can take advantage of the scoping rule...
14:15:31 [plh]
... and substitute it
14:16:29 [plh]
David: some large websites, like Amazon, do detect things like phantomjs already
14:16:38 [plh]
Simon: I'm going to siuggest a header
14:16:49 [plh]
John: we might learn from XHR/CORS
14:17:15 [plh]
... it's sort of a similar problem. the combination of header and attribute could do it
14:18:26 [plh]
... so we must have a fingerprint, with a header and an attribute, and do a security review
14:18:37 [plh]
Simon: let's put a strawman in the spec and get review
14:19:08 [plh]
ACTION: Simon to put a proposal in the spec around the security mechanism header and attribute
14:19:21 [plh]
ACTION: All get their security teams to review Simon's proposal
14:19:46 [plh]
ACTION: Wilhelm to ping the Web Security WG on Simon's proposal
14:20:20 [plh]
John: we have non-modal notification, like saveAs, etc.
14:20:53 [plh]
Simon: we didn't run into those yet, like geolocation. we just defaulted everything to a particular state
14:21:43 [plh]
John: some dialogs have several states, like the activeX (no, never, yes once, yes always)
14:21:59 [plh]
Simon: that's leading in extra capabilities in HTML5...
14:22:28 [plh]
... does user media use the same prompt as geoloc? are those consistent?
14:22:42 [plh]
David: we're consistent
14:22:45 [plh]
John: as we are
14:23:00 [plh]
David: we call those door hangers
14:23:12 [plh]
[David shows the design]
14:23:29 [plh]
[one button with several options]
14:24:44 [plh]
Simon: so, what do we do? provide an API to allow a capability?
14:24:57 [plh]
... we nuke the profile everytime in Chrome
14:25:10 [plh]
... so always share doesn't really matter
14:26:05 [plh]
... we could override switchTo, like switchTo the permission dialog and then use the existing command (dismiss, ok)
14:26:41 [plh]
... if you ignore the dialog, it will default to No
14:27:43 [plh]
Wilhelm: when I test the geoloc spec as a browser vendr, I'd want the four options but that's only 5 orgs in the world
14:28:04 [plh]
Simon: you can set a profile...
14:28:35 [plh]
Simon: so we want to pick between the two accept states
14:28:42 [plh]
Marc: really, it's all four
14:28:52 [plh]
... so it's its own alert object
14:29:13 [plh]
... if we change the wire protocol to allow an index for dialog...
14:29:46 [plh]
Simon: so alert response is the end point?
14:29:51 [plh]
All: yes
14:30:28 [plh]
ACTION: Simon to change the wire protocol to add the alert response and could take 4 responses (yes, yes always, no, no, always)
14:30:32 [plh]
John: what about ignore?
14:30:50 [plh]
... it's functionally equivalent to No
14:30:54 [plh]
Wilhelm: so 4 options
14:31:07 [plh]
John: I still would like to test this...
14:31:17 [plh]
Marc: what about a string with predefined values?
14:31:34 [plh]
Simon: I'm fine with mandating four and allow extensions
14:31:43 [plh]
John: I still would like to have ignore
14:31:53 [AutomatedTester]
Ms2ger: I just showed a doorhanger :)
14:32:09 [plh]
Simon: if we find it's useful, we can always come back to it in version 1.1
14:32:54 [plh]
Simon: a few things to consider now: mocking locations, for image capture, mimicking the data
14:32:59 [plh]
Marc: orientation...
14:33:51 [sstewart6]
Let's get coffee!
14:34:35 [plh]
rrsagent, generate minutes
14:34:35 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html plh
15:01:42 [chrisgao]
chrisgao has joined #testing
15:01:52 [kkania]
Scribe: kkania
15:02:01 [sstewart6]
sstewart6 has joined #testing
15:02:07 [sstewart6]
Present+ SimonStewart
15:02:13 [sstewart6]
ok
15:03:34 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/location
15:04:07 [kkania]
Topic: handling location, camera, and other sensors
15:04:49 [kkania]
sstewart6: we have a number of options to do here
15:04:57 [kkania]
location, and media capture are probably the big 2
15:05:05 [kkania]
that would solve must individual use cases right now
15:05:20 [kkania]
JohnJansen: there is a slightly different question
15:07:22 [kkania]
sstewart6: it sounds like we should ask the people writing the specs to tell us what the best way to supply the mock data is
15:07:34 [sstewart6]
plh: there will be a cat picture
15:07:47 [kkania]
AutomatedTester: we can mock geolocation in js
15:08:10 [sstewart6]
kkania: the link above ponts at the wire protocol
15:08:48 [kkania]
sstewart6: I'm suggesting the two we should bake in now are geolocation and media capture
15:09:00 [kkania]
plh: what about device orientation?
15:09:09 [kkania]
sstewart6: and device orientation
15:09:17 [kkania]
sstewart6: so there's three we should bake
15:09:46 [plh]
http://dev.w3.org/geo/api/spec-source-orientation.html
15:10:18 [sstewart6]
Orientation as it is now: https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/orientation
15:10:39 [kkania]
fisherii: portrait/landscape is different from device orientation
15:11:51 [kkania]
sstewart6: ok, let's go back to our original two, and allow users to specify whether it is upright or landscape
15:12:26 [kkania]
gdennis: there's portrait, landscape, portrait secondary, and landscape secondary
15:13:20 [gdennis]
http://www.w3.org/TR/screen-orientation/
15:13:53 [kkania]
fisherii: we should base orientation off of this spec
15:14:56 [sstewart6]
http://www.w3.org/TR/screen-orientation/#allowed-orientations
15:15:01 [kkania]
JohnJansen: I don't know if we support all 4 states
15:15:10 [kkania]
gdennis: that could be a no-op
15:15:27 [kkania]
fisherii: it could depend on the browser and device
15:15:39 [kkania]
fisherii: does the browser actually support that rotation, or device
15:15:45 [kkania]
but i think it should throw an exception if it is unsupported orientation
15:16:02 [kkania]
gdennis: i don't think it should be an exception, a user can put it in the orientation
15:16:11 [kkania]
fisherii: but screen orientation is not the same as the device orientation
15:16:29 [kkania]
AutomatedTester: so if we were to do all of this, telling the browser to change the orientation is going to be extremely difficult
15:16:41 [kkania]
so in the example of ff os, we can do it on the emulator, but not on real devices
15:17:01 [kkania]
AutomatedTester: we can't get the device to change orientation programmatically
15:17:04 [kkania]
sstewart6: well you could
15:17:19 [kkania]
sstewart6: there are things we'd like to say in the spec, even if the browser can't currently do it
15:17:39 [kkania]
AutomatedTester: can we expect the browsers to do this?
15:17:52 [kkania]
fisherii: well it's capability based, whether controlling the orientation is possible
15:18:06 [kkania]
if the device doesn't support it, the capability should return false
15:18:19 [kkania]
wilhelm: so what does happen if you do turn the device in question, does the orientation change
15:18:30 [kkania]
wilhelm: if the device can actually do it, we should be able to automate it
15:19:14 [kkania]
sstewart6: the language i want in the spec, is if you support a capability, you must provide the functionality
15:20:53 [kkania]
sstewart6: so are we happy with orientation, the four orientation values, for geolocation, latitude, longitude, altitidue
15:20:58 [kkania]
how long do these things stick for
15:21:01 [kkania]
fisherii: until change?
15:21:08 [kkania]
fisherii: rest of the session basically
15:21:10 [kkania]
sstewart6: ok
15:21:42 [kkania]
sstewart6: for media capture, we just allow people to upload a screenshot or something?
15:21:54 [kkania]
AutomatedTester: that's interesting, since you can do video and stuff
15:22:00 [kkania]
AutomatedTester: for webrtc, that's the biggest use case
15:22:06 [kkania]
fisherii: but the video could be a still picture
15:22:13 [kkania]
fisherii: and we don't do that well on dynamic things anyways
15:22:21 [kkania]
AutomatedTester: that's fine, as long as that is documented
15:22:33 [sstewart6]
http://www.w3.org/TR/2012/WD-html-media-capture-20120529/
15:22:39 [kkania]
wilhelm: we could maybe make it simpler by not needing a picture from the user, but by just coloring the screen example
15:23:08 [kkania]
fisherii: camera is easy also, since it is only a single image
15:24:37 [sstewart6]
http://www.w3.org/TR/mediacapture-streams/
15:24:57 [kkania]
fisherii: all of these are just file pickers on some sense
15:25:11 [kkania]
sstewart6: yes, they all upload a file of some sort, but how they capture is differnet
15:25:53 [kkania]
fisherii: so we don't have to do anything special about these, just treat them like any other file
15:25:59 [kkania]
sstewart6: ok, that makes life a lot easier
15:26:10 [kkania]
fisherii: i was thinking when we were talking about media we were talking about media streams
15:26:21 [kkania]
sstewart6: i don't think we're ready to tackle that one yet
15:28:22 [wilhelm]
Photo demo: http://shinydemos.com/photo-booth/
15:29:46 [kkania]
Resolution: we'll support geolocation and screen orientation and handle media capture as standard file inputs; punt on everything else
15:30:30 [kkania]
Topic: Local storage and app cache and normal html caches
15:30:33 [sstewart6]
https://code.google.com/p/selenium/issues/detail?id=40
15:31:25 [kkania]
sstewart6: this is one of the oldest selenium bugs
15:31:28 [kkania]
fisherii: you can clear the cookies now
15:31:39 [kkania]
sstewart6: so that's interesting, since we can't clear the httponly cookies
15:31:56 [kkania]
sstewart6: so, clearing caches, can we do it, should we allow it
15:33:09 [kkania]
sstewart6: at google, people often wanted pre-warmed app caches so when they hit the site they could just start doing things
15:33:15 [kkania]
instead of going through all the hoops
15:33:26 [kkania]
fisherii: so they wanted to start with a known state
15:33:32 [kkania]
can't you do that in ff by serializing the profile
15:34:25 [kkania]
sstewart6: so there's three different caches, html and cookie cache, app cache, and local storage
15:34:40 [kkania]
plh: what about index db?
15:34:52 [plh]
s/index/indexed/
15:34:53 [kkania]
sstewart6: and indexeddb
15:35:22 [kkania]
fisherii: i assumes this is because the don't want to incur overhead of restarting the browser
15:35:34 [kkania]
sstewart6: or if they're using IE, which doesn't use a clean profile per run
15:35:51 [kkania]
sstewart6: or some mobile browsers
15:36:05 [kkania]
fisherii: I would find this useful personally, if we could nuke all of that state
15:38:11 [kkania]
plh: what about clearing security state or geolocation ...?
15:40:27 [kkania]
sstewart6: ok, so we don't support clearing caches midway thru, but we guarantee on start up things are in a clean state
15:41:40 [kkania]
fisherii: except IE can't guarantee clean state
15:41:44 [kkania]
sstewart6: the spec should aim high
15:42:07 [kkania]
John: you can tell IE to clean the state on exit
15:42:46 [kkania]
fisherii: ok, that's good; although there's still difficulty on mobile
15:43:48 [kkania]
sstewart6: should it be a should or a must
15:43:58 [kkania]
John: I think it should be a should
15:44:13 [kkania]
AutomatedTester: should be a must, but pragmatically a should
15:44:32 [kkania]
sstewart6: let's start with a must right now, and we'll have a chance for review
15:45:24 [kkania]
plh: is there a way to test this is implemented
15:45:35 [kkania]
sstewart6: yes, there is, load a file, delete it, and try to access it again
15:46:21 [kkania]
sstewart6: or we could stick a proxy in the way, start a new session, configure with a proxy, request a file, and then request it again, the first time with proper caching headers; restarting the browser, it should download the file again
15:46:40 [kkania]
plh: or you can just do local storage
15:46:44 [kkania]
fisherii: we want tests for all these aspects of state
15:46:53 [kkania]
fisherii: when we talk about the state, we should define what we mean
15:46:59 [jhammel]
jhammel has joined #testing
15:47:03 [jhammel]
jhammel has left #testing
15:47:11 [kkania]
sstewart6: i think you're right, anyone disagree?
15:48:02 [kkania]
Resolution: we're not going to support clearing caches midway through test, we will attempt to start browsers in a clean state, and we will define clean state, and we will write tests around these
15:48:08 [kkania]
plh: that's for clearing, what about setting
15:48:31 [kkania]
sstewart6: let's follow what we do for cookies now, you have to go to the domain
15:48:58 [JohnJansen]
JohnJansen has joined #testing
15:49:20 [kkania]
sstewart6: it might be nice, but it's not a must
15:49:53 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/session_storage
15:50:04 [kkania]
fisherii: there is a spec in open source for local storage/session storage
15:50:25 [kkania]
sstewart6: i think just leave this in the open source spec, until there's a need for it
15:51:01 [plh]
[lunch break]
15:51:01 [kkania]
RRSAgent, draft minutes
15:51:01 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html kkania
16:22:09 [sstewart6]
sstewart6 has joined #testing
16:23:00 [kkania]
kkania has joined #testing
16:30:58 [sstewart6]
scribe sstewart6
16:31:10 [sstewart6]
scribe:sstewart6
16:31:15 [sstewart6]
scribe: SimonStewart
16:31:23 [sstewart6]
scribe: gosh darn it, let me write
16:32:03 [sstewart6]
ScribeNick: sstewart6
16:32:08 [wilhelm]
RRSAgent, draft minutes
16:32:08 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html wilhelm
16:32:17 [mdyck_]
mdyck_ has joined #testing
16:32:20 [sstewart6]
Topic: naming of log nodes
16:33:07 [sstewart6]
How do we prevent collisions in log type names?
16:33:21 [sstewart6]
wilhelm: is it a real problem?
16:33:28 [sstewart6]
It's probably a problem on grid
16:33:42 [sstewart6]
https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html#logging
16:34:06 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/log/types
16:34:15 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#Log_Type
16:34:32 [sstewart6]
"server" is overloaded
16:35:52 [sstewart6]
Options: namespace log types, or allow intermediate nodes to rewrite type names
16:36:01 [sstewart6]
JohnJansen: Why not use namespaces?
16:36:05 [sstewart6]
plh: what would be wrong?
16:36:20 [sstewart6]
wilhelm: mentions xml namespaces in less than glowing terms
16:36:21 [plh]
s/wrong?/wrong? :)/
16:36:45 [sstewart6]
wilhelm: suggests namespacing by adding some unique key from the machine
16:38:36 [sstewart6]
wilhelm: so why not use unique keys based on the host?
16:38:57 [sstewart6]
Because the purpose of the logs is to provide consistent diagnostic data. Hard to do with string parsing
16:39:20 [sstewart6]
Perhaps add "name" and "host" to the log entry
16:39:32 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/log/types
16:40:07 [sstewart6]
https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/log
16:41:34 [sstewart6]
gdennis: Why do you need to find out the types of logs available to you?
16:41:47 [sstewart6]
Because you may not know which logs are available to you
16:41:55 [sstewart6]
gdennis: do the names contain meaning?
16:42:47 [sstewart6]
They can be arbitrary.
16:46:41 [sstewart6]
Conversation about what the purpose of the names are
16:48:18 [sstewart6]
gdennis: the "all" case needs to be handled, but could be done separately
16:51:01 [sstewart6]
JohnJansen: say I get back the strings and we've implemented this. I'd still do the "all" query if I wasn't sure which server was wanted
16:52:26 [sstewart6]
Conversation about highlighting different log types
16:52:35 [sstewart6]
(in a tool)
16:53:25 [sstewart6]
gdennis: you'd be okay if there was a distinctive log type called "all" since the type is in each entry
16:53:49 [sstewart6]
gdennis: not sure the "types" url is necessary
16:54:30 [sstewart6]
JohnJansen: as long as the log itself has a type, I'm not sure that the log types are necessary
16:55:16 [sstewart6]
Discussion about timestamps
16:56:29 [sstewart6]
http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/logging/LogCombiner.html
16:56:52 [sstewart6]
kkania: you could have the browser and performance logs interleaved, provided everything was done on the same machine
16:57:20 [sstewart6]
gdennis: how do we prevent collision of log names
16:57:52 [sstewart6]
gdennis: add the ability to request a log name with a given prefix
16:57:59 [sstewart6]
JohnJansen: and that would be the namespace?
16:58:07 [sstewart6]
gdennis: possibly
16:58:38 [sstewart6]
kkania: but what about the case where there are two types of the same server on different machines? How would you distinguish?
16:59:17 [sstewart6]
JohnJansen: the log types don't have meaning, so you'd need to go through them and post-process
16:59:28 [sstewart6]
kkania: how would the client do the Right Thing for the user
16:59:42 [sstewart6]
gdennis: what's the format of a log type?
17:00:00 [sstewart6]
kkania: not defined. Normally just writing to some memory or a file
17:00:10 [sstewart6]
kkania: chromedriver tells chrome to log to a file
17:00:18 [sstewart6]
gdennis: what's the log type there?
17:00:22 [sstewart6]
kkania: browser
17:00:28 [sstewart6]
gdennis: where do you say that?
17:00:35 [sstewart6]
kkania: the client would request that
17:02:35 [sstewart6]
How do I request all "performance" logs?
17:03:55 [sstewart6]
Suggestion: three pieces of data needed to be sufficiently unique: name of log type, hostname/identifier, and server type
17:04:52 [sstewart6]
kkania: what about the case where there are two identical servers on the same host?
17:05:00 [sstewart6]
How about adding port to the hostname?
17:05:04 [sstewart6]
JohnJansen: that might help
17:05:18 [sstewart6]
In that case we don't need the server type
17:05:47 [sstewart6]
Resolution: add hostname and port to LogEntry
17:06:46 [sstewart6]
kkania: LogEntry is currently time, level, message
17:11:28 [sstewart6]
Discussion about memory of servers
17:12:20 [sstewart6]
JohnJansen: okay to add type and host identifier
17:12:39 [sstewart6]
and data
17:12:44 [sstewart6]
kkania: and data
17:13:37 [sstewart6]
JohnJansen: "type" is about disambiguation
17:14:25 [sstewart6]
kkania: is there a way for the user to request logs from one particular server?
17:14:45 [sstewart6]
kkania: the suggested approach wouldn't allow them to be aware of the types available?
17:15:13 [sstewart6]
No, they wouldn't be aware of the log types available to them
17:15:39 [sstewart6]
kkania: is it important that in the spec that the user has a way to determine the order of the intermediary nodes?
17:18:04 [sstewart6]
what about ordering the log entries, with remote end first and client end last
17:23:31 [sstewart6]
Conversation about ordering of logs
17:24:12 [sstewart6]
gdennis: you could have a "give me an ordering of the sources"
17:24:20 [zcorpan]
zcorpan has joined #testing
17:24:44 [sstewart6]
gdennis: a "source" described by hostname and port.
17:25:19 [sstewart6]
gdennis: there's no way in which logs of one type come before logs of another type from the same source?
17:26:06 [sstewart6]
gdennis: could have "log/sources", which would return the hostnames and ports in order
17:27:52 [sstewart6]
what about the case where you don't want a particular type of log?
17:28:13 [sstewart6]
gdennis: such as "I want all the logs, but not the performance logs?"
17:28:14 [sstewart6]
yes
17:29:44 [sstewart6]
kkania: what about being able to set verbosity of logging of individual nodes?
17:30:53 [sstewart6]
kkania: how about only logging just the remote end?
17:31:11 [sstewart6]
We added logging to allow determining where flakiness in infrastructure was occuring
17:35:40 [sstewart6]
Are you suggesting, kkania, dropping logging entirely?
17:35:51 [sstewart6]
kkania: no, just collecting logs from the most remote end
17:36:24 [sstewart6]
JohnJansen: a browser automation spec need not worry about the components that the automation goes through
17:36:30 [sstewart6]
gdennis: can optionally provide it, though
17:37:04 [sstewart6]
gdennis: it's okay to provide information on sources (hostname and port)
17:37:20 [sstewart6]
gdennis: and you can then request logs from a particular source
17:37:38 [sstewart6]
gdennis: perhaps with an optional type for that source
17:47:26 [abarsto]
abarsto has joined #testing
17:48:25 [sstewart6]
Resolution: two end points for logging. First: "/log/:type/:source" where type and source are optional will return entries for that type and source, allowing either or both to be wildcards.
17:49:00 [sstewart6]
Resolution: Second end point: "/log/available" (or similar): provides a list of sources and types.
17:49:47 [sstewart6]
Resolution: Wire format looks like "{source: {type: [entries], type2: [entries]}, source2: {type: [entries]}}"
17:50:04 [sstewart6]
gdennis: proposal isn't backward compatible?
17:50:09 [sstewart6]
It's not
17:50:54 [sstewart6]
s/are optional//
17:51:25 [sstewart6]
Resolution: "source" is defined as "hostname + port"
17:53:05 [sstewart6]
gdennis: the "excluding" case is the one that means that the "available" end point is needed
17:53:37 [sstewart6]
sources need to returned in order
17:56:30 [JohnJansen]
s/throws something at/throws something heavy at
17:56:50 [sstewart6]
s/something heavy/something very heavy/
18:03:49 [plh]
rrsagent, generate minutes
18:03:49 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html plh
18:04:10 [plh]
</meeting>
18:04:13 [plh]
rrsagent, generate minutes
18:04:13 [RRSAgent]
I have made the request to generate http://www.w3.org/2013/06/14-testing-minutes.html plh
18:04:38 [wilhelm]
RRSAgent, bye
18:04:38 [RRSAgent]
I see 13 open action items saved in http://www.w3.org/2013/06/14-testing-actions.rdf :
18:04:38 [RRSAgent]
ACTION: Simon to figure testing for chapters 9 and 11 [1]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-36-41
18:04:38 [RRSAgent]
ACTION: David to figure testing for chapters 10, 14 and 17 [2]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-37-01
18:04:38 [RRSAgent]
ACTION: Chris to figure testing for chapters 2 and 3 [3]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-37-11
18:04:38 [RRSAgent]
ACTION: Wilhelm to figure testing for chapters 16 [4]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-37-20
18:04:38 [RRSAgent]
ACTION: Chris to figure testing for chapters 15 [5]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-37-36
18:04:38 [RRSAgent]
ACTION: Ken to figure testing for chapters 5 and 6 [6]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-38-02
18:04:38 [RRSAgent]
ACTION: John to figure out testing for chapter 1 [7]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-38-26
18:04:38 [RRSAgent]
ACTION: Simon to create the web-platform-tests directory and move tests there [8]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-39-38
18:04:38 [RRSAgent]
ACTION: David to double check there is no need to extend WebDriver because of Firefox OS [9]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T13-49-21
18:04:38 [RRSAgent]
ACTION: Simon to put a proposal in the spec around the security mechanism header and attribute [10]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T14-19-08
18:04:38 [RRSAgent]
ACTION: All get their security teams to review Simon's proposal [11]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T14-19-21
18:04:38 [RRSAgent]
ACTION: Wilhelm to ping the Web Security WG on Simon's proposal [12]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T14-19-46
18:04:38 [RRSAgent]
ACTION: Simon to change the wire protocol to add the alert response and could take 4 responses (yes, yes always, no, no, always) [13]
18:04:38 [RRSAgent]
recorded in http://www.w3.org/2013/06/14-testing-irc#T14-30-28