Browser testing & tools WG, F2F, July 8th

08 Jul 2014


See also: IRC log


AndreasTolfsen, DavidBurns, JohnJansen, LukeInman-Semerau, MaliniDas, ShuotaoGao, SimonStewart, jgraham, jimevans, wilhelm, MarcFisher
ato, jimevans, JohnJansen, wilhelm


<ato> ty

<wilhelm> Scribe: ato

<simons> https://www.w3.org/wiki/WebDriver/2014-July-F2F

wilhelm: We went through the open bug list yesterday.

Topic: Discuss security issue: navigating to file:/// exposes the whole file system.

Discuss security issue: navigating to file:/// exposes the whole file system.

ShuotaoGao: Accessing file:// protocol has securi issues.

simons: Calling it a bug, is a bit strong. It's working as intended.
... Option to blacklist a protocol, series of URLs, certain blocks might be what you want?

ShuotaoGao: Chrome has whitelist for who can connect to the browser.
... Not for content.

simons: So blacklisting URL schemes?

ShuotaoGao: Yeah, sounds good to me.
... Not sure what others think?

simons: Will IE flip out on file://?

jimevans: Yes, it's not an HTML document.

jgraham: file:// is very underspecified. In some sense there's actually no standard behaviour for you to depend on in this case.
... But does anyone have a use case for file URLs?

MarcFisher: It's something that's done occasionally at Google.
... Different browsers behave differently.
... We tell them to bring up a small HTTP server.
... So it's not a stopping issue.

simons: It's not something I feel we can put normative language around.
... Also what are the use cases, also there are security issues.
... Second of all, the behaviour differs between browsers.
... With IE, the way that it's implemented, it's just not possible to do.
... And on Chrome you can traverse through the file system.
... Normative text in spec on blacklisting URLs?
... Would that be a sensible thing to do?

jgraham: I think it would go in the bit where you define get or whatever it is.
... You'd have a series of steps, and a step where you hit the blacklist, then return an error.

simons: It would not be in the appendix?

jgraham: I'm not sure I believe in that sort of things.

MarcFisher: He's suggesting a non-normative statement. Something you want to think about.

jgraham: Well you can still make that part of the algorithm in get.

simons: click would need to do the same check.

ato: It relates to everything. You can give the URL when starting the browser.
... So you'd need to do the check for literally every call to the WebDriver server.

MarcFisher: Really the file system should be handled by whatever security mechanism is built in to the browser.

JohnJansen: WebDriver is designed to do this.
... The spot where you downloaded it should say “don't forget, this is an incredible security risk”.
... I would never put this on my machine.
... People could guess that I'm using it, and hack my machine.

simons: Neither would I.
... Having the dirvers be configurable.
... WebDriver as a service seems like a use case.
... You don't want to allow them to navigate the file system.

MarcFisher: I don't think on the WebDriver level this is feasible.
... It should be up to people at what level they want security.

jgraham: Yeah, it sounds like it would need a hook in the equivalent of the navigator element in the browser.

MarcFisher: Presumably the HTML5 navigate algorithm already has a hook.
... If they don't, it seems like a bad design.

jgraham: If there isn't an apropriate hook, you can say to Hixie that if this is a URL that has been disallowed for navigation, then abort or something.

MarcFisher: Regardless whether algorithm has that or not, I don't think we can uphold that from WD.

simons: []
... [Explain]

jgraham: If you have a hook in the navigate algorithm you can read the specification and everything makes sense.
... What I'm saying is that Hixie should have a hook, which you hook your spec into, and use this in the WD algorithms.
... Then in your Security appendix you can refer to this.

simons: Yes.
... I don't think this is something we can do entirely in WebDriver.

<scribe> ACTION: jgraham to Talk Hixie (HTML5 guys) about adding some hook into navigate algorithm to prevent navigation to certain subsets of URLs/schemas/protocols. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action01]

JohnJansen: There's also some information online about treating the security for a web server in general.
... On MSDN we have pretty robust docs on this.
... MSDN has a bunch of docs on how to protect yourself from this kind of hack.
... First recommendation: Put the first directory of your server on a separate drive.

<JohnJansen> http://msdn.microsoft.com/en-us/library/ff648653.aspx#c16618429_019

simons: Hurrah!

<JohnJansen> this is different from the file:// protocol, but interesting information regarding the navigation attack.

<scribe> ACTION: To add text in the privacy section that you MAY consider blocking navigation to blacklisted domains or protocols. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action02]

<simons> :)

wilhelm: So that solves the file:// problem, but not connecting to your local machine.

AutomatedTester: We talked about this yesterday, but we removed it.

simons: By default only having it listen on the loopback.

MarcFisher: Might want to have a whitelist of IPs that are allowed to connect.

AutomatedTester: The text I originally put, which is still there…
... Limiting it to localhost would prevent this unless you have a proxy.
... Selenium server could be that proxy.

MarcFisher: You should have a whitelist of IPs, with the default of localhost.

simons: Fine.

<scribe> ACTION: MarcFisher to Change prose around limiting connections in F2 [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action03]

Discuss the test running Microsoft implemented in JScript

JohnJansen: When we were implementing our WebDriver implementation, we needed to run the tests that had already been written in the wpt repository.
... I had an intern convert the Python test suite into a JavaScript test suite so we didn't have to learn or install Python.
... Noone on my team knows Python.
... While my intern was changing everything, he did accidentally learn Python.
... In so doing it, we realized that it is incredibly easy to write tests against WebDriver when you have NodeJS.
... If you want people to contribute to these tests, it seems we want to use JavaScript which everyone knows.

lukeis: Did he use jleyba's bindings?

simons: JohnJansen and tobie makes the point that people who don't work with the W3C are afraid of anything but JavaScript.

jgraham: No that's not true.

AutomatedTester: [agrees]

jgraham: We had this discussion long ago about the wpt repo.
... It used to be a mess.
... Everything is now standardized on Python.

JohnJansen: But the tests are written in JS?

jgraham: No, both.

JohnJansen: So we should write our tests in JS because we're testing WebDriver.

jgraham: My feeling is that if you start depending on NodeJS you're a very special snowflake.

MarcFisher: NodeJS would also mean we'd have to introduce a whole new set of dependencies.

simons: Lack of waiting and sync features.

JohnJansen: We did a hack to NodeJS to make it synchronous.

AutomatedTester: This is what happens everywhere.
... We had this in Mozilla as well. Everyone is doing all these things.

<gitbot> [13web-platform-tests] 15Ms2ger closed pull request #188: new WebGL arguments test files from TestTWF (06master...06submissions/hiroshik850223) 02https://github.com/w3c/web-platform-tests/pull/188

AutomatedTester: Python is easy to read, is a sync language, and is easy to write.
... It's step-by-step.
... People having to put in hacks to make it work with JavaScript, which is the word you used, seems wrong.

JohnJansen: I don't mind if the driver and the runner written in Python.
... But to have the _tests_ written in JavaScript.

MarcFisher: Why?

JohnJansen: It's easier for me to write JavaScript?


JohnJansen: I was trying to do a pull request, but our fork is in a bad state.

wilhelm: Is the fork available anywhere?

simons: Jonathan Lipps did a version of the Sauce JS WebDriver bindings using the ES6 yield keyword.

<simons> https://github.com/jlipps/yiewd

<JohnJansen> https://github.com/tepot2/JST

<simons> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators#Generators.3A_a_better_way_to_build_Iterators

lukeis: I need a test that has cross-domain frames, and you need to have a server on multiple ports or multiple domain names.

JohnJansen: We have internal servers for that.

jgraham: wpt already has this.
... If you run it on your local machine, you do need to edit your /etc/hosts file, but it gives you a big set of cross domain options.

ato: [explains about a patch he's working on for wptrunner]

simons: JS would be kind of nice, but the overhead of learning Python is relatively low.

AutomatedTester: We have a data point from China, where people actually did learn enough Python to contribute.

MarcFisher: Yeah, and then it's just the WebDriver calls which looks very similar.

seva: In JS they don't look as nice.

MarcFisher: Your intern did a good job of turning it into a sync test.
... But the approach is ugly.
... But tests nice.

jgraham: You can't rely on crowd sourcing to get good tests..
... You have to rely on people who know their stuff and have a slight inclination towards evil to sit down and do this.

simons: Yes.

wilhelm: Yes.

<Ms2ger> Yes.

wilhelm: W3C should standardize on our format for writing browser automation tests.
... I want us to keep us to keep this in mind.
... We should figure out what the cost is in labour to use each of these two approaches.
... What are our real alternatives in a five year perspective, and who is going to do the future work here?

AutomatedTester: All the major browser vendors, with the exception of Microsoft, has Python as their major language.
... Their build environments, test environments rely heavily on Python.

ato: In-browser JS is different to NodeJS. It's a whole other beast.

jgraham: I'm certainly against adding more dependencies to wpt.
... And adding Node.js is certainly adding a big dependency.
... It's something new, and it means we have to make sure it's on all our test machines, &c.
... And you already have to know Python to write the server side part of the tests.
... Then you need to learn a new technology, Node.js, to write the tests.

simons: Can we please marshal the arguments I'm hearing?
... [repeats all points above]

jgraham: Node.js is not quite the same as client-side JS.

<gitbot> [13web-platform-tests] 15Velmont closed pull request #116: Start of port of WebKit CORS tests (06master...06nokia_cors_tests) 02https://github.com/w3c/web-platform-tests/pull/116

wilhelm: So there are valid points for each side.
... What are the costs?

JohnJansen: Any new tests we write would have to be converted to Python.

MarcFisher: But that's not very much work?

JohnJansen: There are about 200 tests.
... In this repo I think there's an additional 30 or so tests. Then about 150 on his machine.
... The ones that there are up there are mostly really simple.

simons: What are the other WG going to do?

JohnJansen: I don't want WebDriver to be the only Node.js user.

jgraham: I think the CSS WG is eventually going to be in wpt.

MarcFisher: When people talk about writing tests in JS, they tend to mean writing client-side in-browser tests. So saying Python would clearly indicate out of process tests.

JohnJansen: That's an interesting aspect.

wilhelm: So should we gather some data points?
... Write up a set of canonical tests in each language and present them to the WGs?
... Then hear what they say?

jgraham: I don't think you have to focus too hard on the marketing of this, I think it sells itself.

wilhelm: So how to we solve this?

JohnJansen: I think I understand the argument against taking a Node.js dependency.
... I think I'd like to take the logs back to my team to figure out how much it's going to cost us convert the tests.

ato: Even if we go with the Python approach, there's going to be a lot of work. Many of the tests have not been reviewed properly and many are subject to critical review and possibly rewriting.
... With that in mind it would be good to also take the MS-written JS tests into consideration.

<JohnJansen> ACTION: JohnJansen discuss cost of PYTHON test conversion as well as learning PYTHON with team and report results back to the WG [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action04]

<JohnJansen> ACTION: JohnJansen stop shouting PYTHON [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action05]

<simons> http://ironpython.net/

<JohnJansen> I added an agenda item to review the test suite as well

Discuss touch input section of the spec http://lists.w3.org/Archives/Public/public-browser-tools-testing/2014AprJun/0007.html

<simons> http://lists.w3.org/Archives/Public/public-browser-tools-testing/2014AprJun/0007.html

<wilhelm> http://lists.w3.org/Archives/Public/public-browser-tools-testing/2014AprJun/0007.html

JohnJansen: At the last F2F we discussed implementing touch and touch behaviours in WebDriver.
... And I realized afterwards that we were trying to standardize what people to do their browsers with their hands.
... When you touch and move your finger quickly, that should be recognized as a certain thing.
... This is already being standardized in the Pointer Events spec.

<gitbot> [13web-platform-tests] 15alsemenov closed pull request #609: Tests for callback 'created' (06master...06submission/unipro-custom-elements-4) 02https://github.com/w3c/web-platform-tests/pull/609

JohnJansen: So my feeling is that we shouldn't specify this in the WebDriver specification.
... It would allow for a more generic implementation.
... It would allow for devices such as Kinect.
... Additional ones like pen, which have additional attributes.

AutomatedTester: The one argument I have against it is that no two vendors can agree on what those actions are.
... Google and Apple have recently said they're not doing Pointer Events.

JohnJansen: Google is going to do it.

MarcFisher: It's going to make it difficult to write cross browser tests.
... WebDriver for IE would know how to trigger a hover.

JohnJansen: So each driver would have to special case different browser types and OSes.
... It makes a lot more sense for me, regardless what the input is, to be more generic.
... Standardizing “a gesture is this thing” is not something that's good, because it's specified elsewhere.
... The word “gesture” is problematic.

<JohnJansen> http://www.w3.org/TR/pointerevents/

JohnJansen: I don't think we should go and describe these events in a different way than this spec is doing already.

simons: Yeah. There is hillarious things in there though.

JohnJansen: But each browser does that individually too.

simons: The events that the browser sees are under the glass.
... And we're trying to define what happens when you put your finger on top of the glass.
... If you're mirroring the underlying API too closely, you're really defining the events that are triggered underneath the glass.
... Tap.

JohnJansen: Has pretty diverse meaning.

simons: Different machines may have different settings for breaking events with waiting in between.

simons, JohnJansen: [long discussion]

MarcFisher: We're trying to test web applications mostly. And the fact that in browser X a double tap has to be within 500 ms is completely unimportant to that user.
... What they want to be able to do is to say that this is a double tap.

simons: I think you want both kinds.
... Intent of mind that can be captured. Intending to do a double tap, then underneath it does a sleep for n milliseconds.
... Then you want the primitives.
... It would be nice for the wait primitive to take either a specific or a symbolic timeout.

MarcFisher: The use cases for the primitives are far fewer than the higher level actions.

<wilhelm> http://www.w3.org/2014/02/26-testing-minutes.html#item03

JohnJansen: What about putting your finger down and moving it across the page?

MarcFisher: I want to say that this is either a fling or a touch action.

JohnJansen: But it's not standardized.
... So it's impossible to know.

<mdas> simons: updated spec (not yet reviewed) http://malinidas.com/site_media/webdriver-spec.html#methods-12

JohnJansen: With more generic API you can do a lot more devices.

<mdas> and wait was discussed http://www.w3.org/2014/02/26-testing-minutes.html

simons: Discussion has around periods of time.
... But the right question is, do we have the right level of abstraction?

MarcFisher: Baking it into the API is a reasonable thing.

<simons> There are two issues: timing and API abstraction

MarcFisher: We also need to support higher-level actions because the most common use case is to write tests for a web app across multiple browsers.

mdas: Let's say you want to do a hover.

MarcFisher: Flick, long-press already exists.

AutomatedTester: Flick is gone.

JohnJansen: Conversation last time was, how long is a long press?

AutomatedTester: Differs from browser to browser, so you need to delegate it to the internals.

seva: Could you query what the action timings would be?

simons: No, you wouldn't query the driver implementation across the wire.
... You'd tell it what you want.
... But the browser knows, and consequently the driver which is controlling the browser should know.

<simons> http://www.w3.org/2014/02/26-testing-minutes.html

<simons> http://www.w3.org/2014/02/26-testing-minutes.html#action30

<JohnJansen> http://www.w3.org/2014/02/26-testing-minutes.html#action22

<JohnJansen> http://www.w3.org/2014/02/26-testing-minutes.html#action25

mdas: Why was long press going to be removed?

simons: Down and up, wait for a period

mdas: How would the client bindings know what the wait period would be for each browser?

MarcFisher: They would be symbolic.

mdas: It feels like we should delegate that to the browser.
... The browser wouldn't know it.

simons: Constant.
... Mouse.DOUBLE_TAP
... Handfull of well known.

mdas: Maybe pause can take a parameter, context is true?

simons: Mouse down, pause, take constant which would be a symbol, then mouse up. And other side of wire (remote): Hang on, down, see this is symbol then it uses the constant it has programmed, then up.
... If it's an OS specific thing, then look it up. If it's a browser-specific, do an internal lookup.

mdas: What would be the type of a symbol?

simons: It would be a string.

mdas: Makes sense now.

JohnJansen: Working with the pointer events would be better than specificying this.

simons: Yeah.

AutomatedTester: On a phone you mash the screen.

mdas: Escape.
... But it's totally vendor specific.

JohnJansen: Yes.
... It's not just the browser. IE immersive

behaves differently from the desktop IE.

JohnJansen: A generic pointer can do the same thing with a pen as you do with a mouse as you do with touch.
... I think we should allow them to standardize it.
... Then hook onto that effort.
... We shouldn't standardize what a long press is. But we already resolved that by using a symbol.

simons: Does this spec define what constitutes a double tap?

JohnJansen: No.
... But that's something we need to talk to them about.

MarcFisher: They also match the pointer events to mouse events.
... Section 11
... They don't talk about double click, but they do talk about click and context menu.

simons: This seems like a crazy level of abstraction.
... And we know that that's going to ripple out to all these events.

AutomatedTester: Yeah.

MarcFisher: Also it seems incomplete.

wilhelm: So how do we move forward here?

simons: There's bits from the previous proposal that I'm not hearing people disagree with.
... There are bits that are uncontroversial.
... There is some debate around having different mouse, touch, keyboard events.
... Or if we have a single pointer events interface.

MarcFisher: Keyboard not included.

AutomatedTester: I don't want people to mirror the pointer events spec.

simons: Yes.
... So we need a new name which doesn't leave people mistaking the two.
... Keyboard abstraction, Keyboard, is okay.

People agree.

simons: It sounds like we want to do something with mouse-ish events.

JohnJansen: I'm not sure if it needs to be different.

simons: But AutomatedTester wants to avoid using “Pointer Events”.

JohnJansen: That's why I used “Pointer Actions”.

Room going: Aaah.

simons: Keyboard action, pointer action.

ato: Isn't that local end?

simons: Well we want to get to LC.
... MouseAction which extends PointerAction.

seva: But on the wire protocol level there would only be pointer actions.

simons: Okay so let's discuss the primitives then.
... We need to define some symbols.

JohnJansen: Set attribute "tilt" 3 degrees.
... name/value?

simons: name and an array?
... Then you'd be okay if you wanted to give it more parameters.

simons illustrates API on whiteboard.

<mdas> http://malinidas.com/site_media/note1actions.svg

AutomatedTester now tries to illustrate.

[discussion about data representation]

simons: So we definitely have a KeyboardAction
... Then we have a WaitAction
... … would have a property "duration" which is measured in milliseconds.
... KeyboardAction: keydown, keyup
... Keys constants.
... KeyboardAction: keydown, keyup, wait

MarcFisher: PointerAction > KeyboardAction

JohnJansen: No that doesn't make sense.

Current progress:

Pointer: wait(ms int), release(), setattribute(name string, values []object)

KeyboardAction: keydown(key string), keyup(key string)

PointerAction: down(button int), up(button int), move(x, y int, element WebElement), cancel()


<gitbot> [13web-platform-tests] 15Ms2ger pushed 1 new commit to 06master: 02https://github.com/w3c/web-platform-tests/commit/c227d5cd4aa70eb2d638de33518a02c8bc0f68e7

<gitbot> 13web-platform-tests/06master 14c227d5c 15Ms2ger: Remove backup file.

<wilhelm> Scribe: jimevans

<ato> jimevans: Above I posted the situation so far.

simons giving a summary

two tasks remaining

first, how to cancel and reset all user input state

second, how to do a high-level "do what i mean" method

MarcFisher: proposes automatic reset of user state

discussion around how to reset user input state.

MarcFisher: this should be a separate end point on the API

decision reached. send a delete verb to the actions endpoint is a "reset state command"

simons: how do we design using high-level interaction commands?

high-level actions imply clearing of input state before and after action.

wilhelm: suppose i have a sequence where i've left the input state in a certain state. can i stop and inspect the document?

answer is yes, that's what it's designed to do

MarcFisher: json payload should have a sequence id for cross-action sequencing

mdas: you don't need to do that

MarcFisher: simply correlating the arrays element index is error-prone.

discussion of description of json payload for action sequences

LukeInman-Semerau: what if someone sends malformed or invalid JSON for the action?

simons: throws an error in that case

MarcFisher: devices can define what they accept as valid input

keyboard and mouse devices should probably be singleton

tangent on how many fingers are supported by the microsoft surface table product

seva: did we forget about touchpads?

jimevans: for level 1 we are pretending touchpads do not exist.

the API is fully extensible for any other devices.

simons: does the JSON payload for a specific action require a type property?

acceptable types to be defined in this spec: key, mouse, touch

other types can be defined using vendor prefixes.

discussion about how strict the typechecking of attributes in action JSON

specific example: a "tilt" attribute in the JSON object. how do we implement?

additional action names and additional properties in the actions will be handled by vendor prefixes until standardized in later spec levels

<scribe> unknown action types or unknown fields in existing action descriptions will all return an error code.

<scribe> ACTION: mdas to strictly specify all supported device types, action types, and parameters in each action type. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action06]

JohnJansen: can we standardize on pen actions?

simons: once this change has been landed, "pen" input actions can be added

<scribe> ACTION: JohnJansen to patch the spec with pen input actions [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action07]

discussion around which wait times are required for the wait action

<scribe> ACTION: mdas to define symbols for wait times [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action08]

ShuotaoGao: which action should happen first in a multi-action sequence?

as close to simultaneous as possible.

seva: since they can't be actually simultaneous, should we specify the order?

simons: there may be a millisecond between the actions, but it should be close enough.
... once we've got an implementation, since no one has implemented this...

mdas and DavidBurns: we have!

simons: well, not this brand-new implementation, so when that is done, we can see what issues, if any, occur.

JohnJansen: sequence of "simultaneous" actions happen in the order the actions appear in the array.

mdas: we don't control the event queue

simons: no, but you can control what order you put things into the event queue.

MarcFisher: but what if an implementation decides to always process keyboard first, then mouse, for example,

simons: it's still an implementation detail to be left to the discretion of the implementor.

decision: order of actions within the array will not be specified.

<simons> https://www.w3.org/wiki/WebDriver/2014-July-F2F

input elements where type=file and multiple file inputs

Discuss input type=file & multiple

simons: how do we handle input elements with different (think HTML5) inputs?

discussion from previous f2f, which never got added to the spec.

<scribe> ACTION: MarcFisher to add information to the spec about additional types of input elements. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action09]

What will the web-platform-tests webdriver test runner look like/use?

mdas: how will i run the wpt tests that are webdriver tests when i have a driver that uses a tcp socket (not an http transport)?

JamesGraham: the test runner is configurable for different browsers, so we should be able to allow that.

ato: one common api, but separate code will be required for launching each browser
... the life cycle of the browser should be managed by the runner, not the tests themselves.

<AutomatedTester> ShuotaoGao: https://github.com/SeleniumHQ/selenium/pull/168

<JohnJansen> scribe: JohnJansen

seva: test runner should manage browser sessions.
... does it already do that? or need an action?

jgraham: don't need a new action, it's an ongoing effort

<simons> http://www.w3.org/2014/02/25-testing-minutes.html

ato: I'm working on it.

<wilhelm> http://www.w3.org/2014/02/25-testing-minutes.html

<lukeis> http://www.w3.org/2014/02/25-testing-minutes.html#ActionSummary

Current Action items from above link

1. Add definition for getElementText to the appropriate section of spec

simons: already done

2. ato to research root element behaviour in HTML and XML

lukeis: I will create bugs for these

AutomatedTester: bug has been logged against CSSWG

<AutomatedTester> https://www.w3.org/Bugs/Public/show_bug.cgi?id=23825

<simons> MikeSmith is about to get an action item…

<simons> Excellent.

[going over actions from last time and logging bugs]

<simons> http://www.w3.org/2014/02/26-testing-minutes.html#action30

<simons> JohnJansen to investigate setting up a mechanism to determine which requirements in the spec have tests

all Actions from previous F2F have been modified into bugs

as necessary


HTML encoding over the wire

simons: sendKeys() to the date format input type

<seva> seva: raised https://www.w3.org/Bugs/Public/show_bug.cgi?id=26290 and 26291; others were "to write tests" which we decided not to raise and hope for bug 26291 to take care of tracking.

jgraham: go through HTML spec element by element

marcfisher: when you sendKeys() to a field that makes sense to append, we append. With color (for example), we replace.
... I have an action to update the spec with this language

JimEvans: that answers one question. #2 are we considering using low level actions as the basis for higher level actions.

<selbot2> 03

<selbot2> dawagner@gmail.com closed/Fixed DOCS: Typo in PageFactory documentation - https://code.google.com/p/selenium/issues/detail?id=2 [Type-Defect ]

simons: you should be able to model them, but not required

are we on course for our proposed timeline?

<wilhelm> http://www.w3.org/2011/08/browser-testing-charter.html

wilhelm: we have a charter to 31 Dec 2015
... milestones, however need to be met:

ato: only AutomatedTester is editing the spec, and he seems too busy to meet the deadlines

jgraham: I think one person owning it makes sense, though, because he doesn't need to wait for others. On the other hand, having that same person write all of the tests doesn't make sense.

it's difficult to know what time I'll have over the coming months

simons: hard to carve out time, but if we schedule meetings, it might work well.

jgraham: CSSWG has done a good job jointly writing specs, but that is the rare example.

ato: unrealistic that David can do this all himself. Perhaps mdas could take some actions from David.

<gitbot> [13web-platform-tests] 15hayatoito closed pull request #899: shadow dom: add code confirm activeElement is readonly and refactor the ... (06master...06submissions/iseki-masaya.activeElement) 02https://github.com/w3c/web-platform-tests/pull/899

jgraham: makes sense to distribute draft material, and then have one editor


[discussion of recurring spec hackathons]

mdas: what are the major parts we need to draft?

AutomatedTester: there aren't a lot of new parts, but we need to rework a lot of the prose (start from scratch in certain sections)

wilhelm: james, can you help write draft patches?

AutomatedTesting: james's focus right now is on the testing, which is where we need him.

Wilhelm: TPAC is in Oct 30 and 31st

<simons> GTAC is 28-29 Oct

<simons> Starwest is 12-17 Oct


so if we're going to be Last Call before TPAC, we need to schedule work

ato: as we write tests we will find issues in the spec
... I've been writing tests and I will take the spearhead on the test suite

JohnJansen: I will work to see if the CSSWG process will work with our spec

jgraham: fair warning: it is easy to fall into a trap of just testing the spec

I agree with James. we need ot meet W3C REC requirements but ALSO test the spec, the most interetsting bugs will be combinatorial in nature.

wilhelm: will MS have resources to help?

Yes, but not sure I can convert to Python as well

I'll have my intern make them public even if just Jscript versions to help.

jgraham: some cool metaphor about bugs being flies that safely sail through spider webs of boring test cases

wilhelm: lukeis will you have time?

lukeis: likely, yes. Same with JimEvans
... some time can be carved out.

wilhelm: so we are carving out test time, next the spec authors need to get together to work on the spec itself

i'll put all of our tests up in git by 18 July.

mdas: I'll work on it this week

<wilhelm> Scribe: wilhelm

Review of the test suite

<gitbot> [13web-platform-tests] 15sideshowbarker closed pull request #325: app-URI spec compliance tests (06master...06submission/martap) 02https://github.com/w3c/web-platform-tests/pull/325

ato: The Python test suite is very unorganized, so it'll take a long while for all of us to go through it together.

JohnJansen: So we should look through it individually?

ato: All the tests require critical review.
... The spec has also changed dramatically since the tests were written.

JohnJansen: When we submit tests, we flag them as being not reviewed?

ato: All the current tests must be reviewed.

JohnJansen: I previously assumed they had been reviewed.

ato: We started with the tests in a hg repo. We have since moved them to wpt.
... Many tests have not been reviewed.
... The majority.
... They even need a re-review now that the spec has changed.

jgraham: This is a perennial problem with writing tests against a spec that is also changing.

ato: My gut feeling is that we should revisit this topic at the next F2F.

jimevans: That isn't until TPAC, though.

ato: What if I go through them, and send out an email?

jgraham: Yes, it makes more sense to use async communication.

ato: Yes, especially code review.

seva: Is this to check that the group is okay with the tests?

Marc: I changed the tests to use the new driver.

ato: One of the sections I'm most unsure about is the visibility section.
... 90% of the tests are not covered by the algorithm.
... SHould we remove those, or add them to the algorithm?

Marc: both.

simons: Yes.

seva: Where are the tests from?

ato: Written by me.

seva: I added some of those.

ato: There's also fresh tests that are not in Selenium.

<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py

JohnJansen: The test with omitted body tag, both Chrome and FF fails.

seva: It tests what the spec says...
... I submitted a fix of this a week or two ago into the atoms.
... ChromeDriver uses old atoms.

JohnJansen: We'll need to choose if we should comply with the spec, or interop.

lukeis: Compliant.

JohnJansen: What we find is that the web often doesn't agree...

seva: With the body element, we found the root cause. Fixed two weeks ago.

wilhelm: If the tests are more detailed than the spec, the behaviour should go into the spec.

JohnJansen: I could just call out the 14 tests that both Chrome or Firefox fail.

jgraham: Once we have wptrunner running, we can get results for multiple browsers in JSON.

JohnJansen: We are making decisions on these tests...

<JohnJansen> https://github.com/w3c/web-platform-tests/tree/master/webdriver

<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py

<JohnJansen> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py#L190

seva: This issue has been fixed.

ato: This only makes sense for HTML, right?
... We should probably have one for HTML and one for XML.

jgraham: By the time you get it, it'll be a DOM tree.

(Discussion on the specifics on this particular test.)

MarcFisher_: This test found a real bug.
... I think it's worth keeping.

seva: We should modify the wording in the spec to not talk about implicit elements?

(Discussion on special behaviour of <body> in HTML vs XML, quirks mode and standards mode.)

<seva> I was wrong, the spec does not actuslly talk about implicit BODY so there's nothing to change in there

ato: Is't this outside the scope of the spec?

JohnJansen: We're trying to define visibility not as defined by CSS, but as seen to a used.

jgraham: The algorithm is using DOM or CSS properties.

<JohnJansen> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py#L131

RESOLUTION: Follow the spec on https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py#L190

<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/res/element-hidden-by-z-index.html

(Discussion on how to detect which element is on top, judging by z-index.)

<scribe> ACTION: JohnJansen to figure out how IE passes https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/res/element-hidden-by-z-index.html [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action10]

<JohnJansen> https://github.com/w3c/web-platform-tests/blob/master/webdriver/navigation/auth_tests.py#L39

JohnJansen: This test fails in Chrome and FF.

lukeis: This is one of the oldest bugs in WebDriver...

ato: Is this in the spec?
... Selenium doesn't support this.

RESOLUTION: We move 401 auth to level 2

<lukeis> #34

<selbot2> 03

<selbot2> simon.m.stewart open/Accepted Support BASIC and Digest HTTP authentication - https://code.google.com/p/selenium/issues/detail?id=34 [Type-Enhancement Priority-Medium Component-WebDriver ]

<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/method_test.py

JohnJansen: The test I'm referring to requests and attribute that doesn't exist. Both FF and Chrome returns an error.

AutomatedTester: They should return null.

<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/res/element-without-attribute.html

MarcFisher__: It's quite possible that our special casing is broken.

(References to different, related tests.)

MarcFisher__: The special casing around class is breaking it somehow.

JohnJansen: The remaining are all the same: six failures because of status code.

simons: Returning strings is correct.

RESOLUTION: Use strings for errors.

(Discussion on compat mode vs legacy mode in the current driver.)

JohnJansen: That's all!

Switch to window

(Discussion of identifier for windows.)

<scribe> ACTION: switchToWindow shall only take window handles [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action11]



Meeting finished!

Summary of Action Items

[NEW] ACTION: jgraham to Talk Hixie (HTML5 guys) about adding some hook into navigate algorithm to prevent navigation to certain subsets of URLs/schemas/protocols. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action01]
[NEW] ACTION: JohnJansen discuss cost of PYTHON test conversion as well as learning PYTHON with team and report results back to the WG [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action04]
[NEW] ACTION: JohnJansen stop shouting PYTHON [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action05]
[NEW] ACTION: JohnJansen to figure out how IE passes https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/res/element-hidden-by-z-index.html [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action10]
[NEW] ACTION: JohnJansen to patch the spec with pen input actions [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action07]
[NEW] ACTION: MarcFisher to add information to the spec about additional types of input elements. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action09]
[NEW] ACTION: MarcFisher to Change prose around limiting connections in F2 [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action03]
[NEW] ACTION: mdas to define symbols for wait times [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action08]
[NEW] ACTION: mdas to strictly specify all supported device types, action types, and parameters in each action type. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action06]
[NEW] ACTION: switchToWindow shall only take window handles [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action11]
[NEW] ACTION: To add text in the privacy section that you MAY consider blocking navigation to blacklisted domains or protocols. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action02]
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2014-07-08 15:11:46 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.138  of Date: 2013-04-25 13:59:11  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/Presnet+ AndreasTolfsen//
Succeeded: s/Can someone come and collect me at reception?//
Succeeded: s/ato: Simon is en route//
Succeeded: s/ty//
Succeeded: s/:yt never going to give you up//
Succeeded: s|Rick Astley - Never Gonna Give You Up - http://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=youtube_gdata||
Succeeded: s|Rick Astley - Never Gonna Give You Up - http://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=youtube_gdata||g
Succeeded: s/blacklisted domains/blacklisted domains or protocols/
Succeeded: s/There's/There are/
Succeeded: s/MarchFisher/MarcFisher/
Succeeded: s/AutomatedTester: Escape/mdas: Escape/
Succeeded: s/ACTION: strictly specify all supported device types, action types, and parameters in each action type./ACTION: mdas to strictly specify all supported device types, action types, and parameters in each action type./
Succeeded: s/wait attribute/wait action/
Succeeded: s/itselfr/itself/
Succeeded: s/mot/most/
Found Scribe: ato
Inferring ScribeNick: ato
Found Scribe: jimevans
Inferring ScribeNick: jimevans
Found Scribe: JohnJansen
Inferring ScribeNick: JohnJansen
Found Scribe: wilhelm
Inferring ScribeNick: wilhelm
Scribes: ato, jimevans, JohnJansen, wilhelm
ScribeNicks: ato, jimevans, JohnJansen, wilhelm
Present: AndreasTolfsen DavidBurns JohnJansen LukeInman-Semerau MaliniDas ShuotaoGao SimonStewart jgraham jimevans wilhelm MarcFisher
Agenda: https://www.w3.org/wiki/WebDriver/2014-July-F2F
Got date from IRC log name: 08 Jul 2014
Guessing minutes URL: http://www.w3.org/2014/07/08-testing-minutes.html
People with action items: as back conversion cost discuss jgraham johnjansen learning marcfisher mdas of python report results switchtowindow team test to well with

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.

[End of scribe.perl diagnostic output]