See also: IRC log
<ato> ty
<wilhelm> Scribe: ato
<simons> https://www.w3.org/wiki/WebDriver/2014-July-F2F
wilhelm: We went through the open bug list yesterday.
ShuotaoGao: Accessing file:// protocol has securi issues.
simons: Calling it a bug, is a
bit strong. It's working as intended.
... Option to blacklist a protocol, series of URLs, certain
blocks might be what you want?
ShuotaoGao: Chrome has whitelist
for who can connect to the browser.
... Not for content.
simons: So blacklisting URL schemes?
ShuotaoGao: Yeah, sounds good to
me.
... Not sure what others think?
simons: Will IE flip out on file://?
jimevans: Yes, it's not an HTML document.
jgraham: file:// is very
underspecified. In some sense there's actually no standard
behaviour for you to depend on in this case.
... But does anyone have a use case for file URLs?
MarcFisher: It's something that's
done occasionally at Google.
... Different browsers behave differently.
... We tell them to bring up a small HTTP server.
... So it's not a stopping issue.
simons: It's not something I feel
we can put normative language around.
... Also what are the use cases, also there are security
issues.
... Second of all, the behaviour differs between
browsers.
... With IE, the way that it's implemented, it's just not
possible to do.
... And on Chrome you can traverse through the file
system.
... Normative text in spec on blacklisting URLs?
... Would that be a sensible thing to do?
jgraham: I think it would go in
the bit where you define get or whatever it is.
... You'd have a series of steps, and a step where you hit the
blacklist, then return an error.
simons: It would not be in the appendix?
jgraham: I'm not sure I believe in that sort of things.
MarcFisher: He's suggesting a non-normative statement. Something you want to think about.
jgraham: Well you can still make that part of the algorithm in get.
simons: click would need to do the same check.
ato: It relates to everything.
You can give the URL when starting the browser.
... So you'd need to do the check for literally every call to
the WebDriver server.
MarcFisher: Really the file system should be handled by whatever security mechanism is built in to the browser.
JohnJansen: WebDriver is designed
to do this.
... The spot where you downloaded it should say “don't forget,
this is an incredible security risk”.
... I would never put this on my machine.
... People could guess that I'm using it, and hack my
machine.
simons: Neither would I.
... Having the dirvers be configurable.
... WebDriver as a service seems like a use case.
... You don't want to allow them to navigate the file
system.
MarcFisher: I don't think on the
WebDriver level this is feasible.
... It should be up to people at what level they want
security.
jgraham: Yeah, it sounds like it would need a hook in the equivalent of the navigator element in the browser.
MarcFisher: Presumably the HTML5
navigate algorithm already has a hook.
... If they don't, it seems like a bad design.
jgraham: If there isn't an apropriate hook, you can say to Hixie that if this is a URL that has been disallowed for navigation, then abort or something.
MarcFisher: Regardless whether algorithm has that or not, I don't think we can uphold that from WD.
simons: []
... [Explain]
jgraham: If you have a hook in
the navigate algorithm you can read the specification and
everything makes sense.
... What I'm saying is that Hixie should have a hook, which you
hook your spec into, and use this in the WD algorithms.
... Then in your Security appendix you can refer to this.
simons: Yes.
... I don't think this is something we can do entirely in
WebDriver.
<scribe> ACTION: jgraham to Talk Hixie (HTML5 guys) about adding some hook into navigate algorithm to prevent navigation to certain subsets of URLs/schemas/protocols. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action01]
JohnJansen: There's also some
information online about treating the security for a web server
in general.
... On MSDN we have pretty robust docs on this.
... MSDN has a bunch of docs on how to protect yourself from
this kind of hack.
... First recommendation: Put the first directory of your
server on a separate drive.
<JohnJansen> http://msdn.microsoft.com/en-us/library/ff648653.aspx#c16618429_019
simons: Hurrah!
<JohnJansen> this is different from the file:// protocol, but interesting information regarding the navigation attack.
<scribe> ACTION: To add text in the privacy section that you MAY consider blocking navigation to blacklisted domains or protocols. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action02]
<simons> :)
wilhelm: So that solves the file:// problem, but not connecting to your local machine.
AutomatedTester: We talked about this yesterday, but we removed it.
simons: By default only having it listen on the loopback.
MarcFisher: Might want to have a whitelist of IPs that are allowed to connect.
AutomatedTester: The text I
originally put, which is still there…
... Limiting it to localhost would prevent this unless you have
a proxy.
... Selenium server could be that proxy.
MarcFisher: You should have a whitelist of IPs, with the default of localhost.
simons: Fine.
<scribe> ACTION: MarcFisher to Change prose around limiting connections in F2 [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action03]
JohnJansen: When we were
implementing our WebDriver implementation, we needed to run the
tests that had already been written in the wpt
repository.
... I had an intern convert the Python test suite into a
JavaScript test suite so we didn't have to learn or install
Python.
... Noone on my team knows Python.
... While my intern was changing everything, he did
accidentally learn Python.
... In so doing it, we realized that it is incredibly easy to
write tests against WebDriver when you have NodeJS.
... If you want people to contribute to these tests, it seems
we want to use JavaScript which everyone knows.
lukeis: Did he use jleyba's bindings?
simons: JohnJansen and tobie makes the point that people who don't work with the W3C are afraid of anything but JavaScript.
jgraham: No that's not true.
AutomatedTester: [agrees]
jgraham: We had this discussion
long ago about the wpt repo.
... It used to be a mess.
... Everything is now standardized on Python.
JohnJansen: But the tests are written in JS?
jgraham: No, both.
JohnJansen: So we should write our tests in JS because we're testing WebDriver.
jgraham: My feeling is that if you start depending on NodeJS you're a very special snowflake.
MarcFisher: NodeJS would also mean we'd have to introduce a whole new set of dependencies.
simons: Lack of waiting and sync features.
JohnJansen: We did a hack to NodeJS to make it synchronous.
AutomatedTester: This is what
happens everywhere.
... We had this in Mozilla as well. Everyone is doing all these
things.
<gitbot> [13web-platform-tests] 15Ms2ger closed pull request #188: new WebGL arguments test files from TestTWF (06master...06submissions/hiroshik850223) 02https://github.com/w3c/web-platform-tests/pull/188
AutomatedTester: Python is easy
to read, is a sync language, and is easy to write.
... It's step-by-step.
... People having to put in hacks to make it work with
JavaScript, which is the word you used, seems wrong.
JohnJansen: I don't mind if the
driver and the runner written in Python.
... But to have the _tests_ written in JavaScript.
MarcFisher: Why?
JohnJansen: It's easier for me to write JavaScript?
JohnJansen: I was trying to do a pull request, but our fork is in a bad state.
wilhelm: Is the fork available anywhere?
simons: Jonathan Lipps did a version of the Sauce JS WebDriver bindings using the ES6 yield keyword.
<simons> https://github.com/jlipps/yiewd
<JohnJansen> https://github.com/tepot2/JST
lukeis: I need a test that has cross-domain frames, and you need to have a server on multiple ports or multiple domain names.
JohnJansen: We have internal servers for that.
jgraham: wpt already has
this.
... If you run it on your local machine, you do need to edit
your /etc/hosts file, but it gives you a big set of cross
domain options.
ato: [explains about a patch he's working on for wptrunner]
simons: JS would be kind of nice, but the overhead of learning Python is relatively low.
AutomatedTester: We have a data point from China, where people actually did learn enough Python to contribute.
MarcFisher: Yeah, and then it's just the WebDriver calls which looks very similar.
seva: In JS they don't look as nice.
MarcFisher: Your intern did a
good job of turning it into a sync test.
... But the approach is ugly.
... But tests nice.
jgraham: You can't rely on crowd
sourcing to get good tests..
... You have to rely on people who know their stuff and have a
slight inclination towards evil to sit down and do this.
simons: Yes.
wilhelm: Yes.
<Ms2ger> Yes.
wilhelm: W3C should standardize
on our format for writing browser automation tests.
... I want us to keep us to keep this in mind.
... We should figure out what the cost is in labour to use each
of these two approaches.
... What are our real alternatives in a five year perspective,
and who is going to do the future work here?
AutomatedTester: All the major
browser vendors, with the exception of Microsoft, has Python as
their major language.
... Their build environments, test environments rely heavily on
Python.
ato: In-browser JS is different to NodeJS. It's a whole other beast.
jgraham: I'm certainly against
adding more dependencies to wpt.
... And adding Node.js is certainly adding a big
dependency.
... It's something new, and it means we have to make sure it's
on all our test machines, &c.
... And you already have to know Python to write the server
side part of the tests.
... Then you need to learn a new technology, Node.js, to write
the tests.
simons: Can we please marshal the
arguments I'm hearing?
... [repeats all points above]
jgraham: Node.js is not quite the same as client-side JS.
<gitbot> [13web-platform-tests] 15Velmont closed pull request #116: Start of port of WebKit CORS tests (06master...06nokia_cors_tests) 02https://github.com/w3c/web-platform-tests/pull/116
wilhelm: So there are valid
points for each side.
... What are the costs?
JohnJansen: Any new tests we write would have to be converted to Python.
MarcFisher: But that's not very much work?
JohnJansen: There are about 200
tests.
... In this repo I think there's an additional 30 or so tests.
Then about 150 on his machine.
... The ones that there are up there are mostly really
simple.
simons: What are the other WG going to do?
JohnJansen: I don't want WebDriver to be the only Node.js user.
jgraham: I think the CSS WG is eventually going to be in wpt.
MarcFisher: When people talk about writing tests in JS, they tend to mean writing client-side in-browser tests. So saying Python would clearly indicate out of process tests.
JohnJansen: That's an interesting aspect.
wilhelm: So should we gather some
data points?
... Write up a set of canonical tests in each language and
present them to the WGs?
... Then hear what they say?
jgraham: I don't think you have to focus too hard on the marketing of this, I think it sells itself.
wilhelm: So how to we solve this?
JohnJansen: I think I understand
the argument against taking a Node.js dependency.
... I think I'd like to take the logs back to my team to figure
out how much it's going to cost us convert the tests.
ato: Even if we go with the
Python approach, there's going to be a lot of work. Many of the
tests have not been reviewed properly and many are subject to
critical review and possibly rewriting.
... With that in mind it would be good to also take the
MS-written JS tests into consideration.
<JohnJansen> ACTION: JohnJansen discuss cost of PYTHON test conversion as well as learning PYTHON with team and report results back to the WG [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action04]
<JohnJansen> ACTION: JohnJansen stop shouting PYTHON [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action05]
<simons> http://ironpython.net/
<JohnJansen> I added an agenda item to review the test suite as well
<simons> http://lists.w3.org/Archives/Public/public-browser-tools-testing/2014AprJun/0007.html
<wilhelm> http://lists.w3.org/Archives/Public/public-browser-tools-testing/2014AprJun/0007.html
JohnJansen: At the last F2F we
discussed implementing touch and touch behaviours in
WebDriver.
... And I realized afterwards that we were trying to
standardize what people to do their browsers with their
hands.
... When you touch and move your finger quickly, that should be
recognized as a certain thing.
... This is already being standardized in the Pointer Events
spec.
<gitbot> [13web-platform-tests] 15alsemenov closed pull request #609: Tests for callback 'created' (06master...06submission/unipro-custom-elements-4) 02https://github.com/w3c/web-platform-tests/pull/609
JohnJansen: So my feeling is that
we shouldn't specify this in the WebDriver specification.
... It would allow for a more generic implementation.
... It would allow for devices such as Kinect.
... Additional ones like pen, which have additional
attributes.
AutomatedTester: The one argument
I have against it is that no two vendors can agree on what
those actions are.
... Google and Apple have recently said they're not doing
Pointer Events.
JohnJansen: Google is going to do it.
MarcFisher: It's going to make it
difficult to write cross browser tests.
... WebDriver for IE would know how to trigger a hover.
JohnJansen: So each driver would
have to special case different browser types and OSes.
... It makes a lot more sense for me, regardless what the input
is, to be more generic.
... Standardizing “a gesture is this thing” is not something
that's good, because it's specified elsewhere.
... The word “gesture” is problematic.
<JohnJansen> http://www.w3.org/TR/pointerevents/
JohnJansen: I don't think we should go and describe these events in a different way than this spec is doing already.
simons: Yeah. There is hillarious things in there though.
JohnJansen: But each browser does that individually too.
simons: The events that the
browser sees are under the glass.
... And we're trying to define what happens when you put your
finger on top of the glass.
... If you're mirroring the underlying API too closely, you're
really defining the events that are triggered underneath the
glass.
... Tap.
JohnJansen: Has pretty diverse meaning.
simons: Different machines may have different settings for breaking events with waiting in between.
simons, JohnJansen: [long discussion]
MarcFisher: We're trying to test
web applications mostly. And the fact that in browser X a
double tap has to be within 500 ms is completely unimportant to
that user.
... What they want to be able to do is to say that this is a
double tap.
simons: I think you want both
kinds.
... Intent of mind that can be captured. Intending to do a
double tap, then underneath it does a sleep for n
milliseconds.
... Then you want the primitives.
... It would be nice for the wait primitive to take either a
specific or a symbolic timeout.
MarcFisher: The use cases for the primitives are far fewer than the higher level actions.
<wilhelm> http://www.w3.org/2014/02/26-testing-minutes.html#item03
JohnJansen: What about putting your finger down and moving it across the page?
MarcFisher: I want to say that this is either a fling or a touch action.
JohnJansen: But it's not
standardized.
... So it's impossible to know.
<mdas> simons: updated spec (not yet reviewed) http://malinidas.com/site_media/webdriver-spec.html#methods-12
JohnJansen: With more generic API you can do a lot more devices.
<mdas> and wait was discussed http://www.w3.org/2014/02/26-testing-minutes.html
simons: Discussion has around
periods of time.
... But the right question is, do we have the right level of
abstraction?
MarcFisher: Baking it into the API is a reasonable thing.
<simons> There are two issues: timing and API abstraction
MarcFisher: We also need to support higher-level actions because the most common use case is to write tests for a web app across multiple browsers.
mdas: Let's say you want to do a hover.
MarcFisher: Flick, long-press already exists.
AutomatedTester: Flick is gone.
JohnJansen: Conversation last time was, how long is a long press?
AutomatedTester: Differs from browser to browser, so you need to delegate it to the internals.
seva: Could you query what the action timings would be?
simons: No, you wouldn't query
the driver implementation across the wire.
... You'd tell it what you want.
... But the browser knows, and consequently the driver which is
controlling the browser should know.
<simons> http://www.w3.org/2014/02/26-testing-minutes.html
<simons> http://www.w3.org/2014/02/26-testing-minutes.html#action30
<JohnJansen> http://www.w3.org/2014/02/26-testing-minutes.html#action22
<JohnJansen> http://www.w3.org/2014/02/26-testing-minutes.html#action25
mdas: Why was long press going to be removed?
simons: Down and up, wait for a period
mdas: How would the client bindings know what the wait period would be for each browser?
MarcFisher: They would be symbolic.
mdas: It feels like we should
delegate that to the browser.
... The browser wouldn't know it.
simons: Constant.
... Mouse.DOUBLE_TAP
... Handfull of well known.
mdas: Maybe pause can take a parameter, context is true?
simons: Mouse down, pause, take
constant which would be a symbol, then mouse up. And other side
of wire (remote): Hang on, down, see this is symbol then it
uses the constant it has programmed, then up.
... If it's an OS specific thing, then look it up. If it's a
browser-specific, do an internal lookup.
mdas: What would be the type of a symbol?
simons: It would be a string.
mdas: Makes sense now.
JohnJansen: Working with the pointer events would be better than specificying this.
simons: Yeah.
AutomatedTester: On a phone you mash the screen.
mdas: Escape.
... But it's totally vendor specific.
JohnJansen: Yes.
... It's not just the browser. IE immersive
behaves differently from the desktop IE.
JohnJansen: A generic pointer can
do the same thing with a pen as you do with a mouse as you do
with touch.
... I think we should allow them to standardize it.
... Then hook onto that effort.
... We shouldn't standardize what a long press is. But we
already resolved that by using a symbol.
simons: Does this spec define what constitutes a double tap?
JohnJansen: No.
... But that's something we need to talk to them about.
MarcFisher: They also match the
pointer events to mouse events.
... Section 11
... They don't talk about double click, but they do talk about
click and context menu.
simons: This seems like a crazy
level of abstraction.
... And we know that that's going to ripple out to all these
events.
AutomatedTester: Yeah.
MarcFisher: Also it seems incomplete.
wilhelm: So how do we move forward here?
simons: There's bits from the
previous proposal that I'm not hearing people disagree
with.
... There are bits that are uncontroversial.
... There is some debate around having different mouse, touch,
keyboard events.
... Or if we have a single pointer events interface.
MarcFisher: Keyboard not included.
AutomatedTester: I don't want people to mirror the pointer events spec.
simons: Yes.
... So we need a new name which doesn't leave people mistaking
the two.
... Keyboard abstraction, Keyboard, is okay.
People agree.
simons: It sounds like we want to do something with mouse-ish events.
JohnJansen: I'm not sure if it needs to be different.
simons: But AutomatedTester wants to avoid using “Pointer Events”.
JohnJansen: That's why I used “Pointer Actions”.
Room going: Aaah.
simons: Keyboard action, pointer action.
ato: Isn't that local end?
simons: Well we want to get to
LC.
... MouseAction which extends PointerAction.
seva: But on the wire protocol level there would only be pointer actions.
simons: Okay so let's discuss the
primitives then.
... We need to define some symbols.
JohnJansen: Set attribute "tilt"
3 degrees.
... name/value?
simons: name and an array?
... Then you'd be okay if you wanted to give it more
parameters.
simons illustrates API on whiteboard.
<mdas> http://malinidas.com/site_media/note1actions.svg
AutomatedTester now tries to illustrate.
[discussion about data representation]
simons: So we definitely have a
KeyboardAction
... Then we have a WaitAction
... … would have a property "duration" which is measured in
milliseconds.
... KeyboardAction: keydown, keyup
... Keys constants.
... KeyboardAction: keydown, keyup, wait
MarcFisher: PointerAction > KeyboardAction
JohnJansen: No that doesn't make sense.
Current progress:
Pointer: wait(ms int), release(), setattribute(name string, values []object)
KeyboardAction: keydown(key string), keyup(key string)
PointerAction: down(button int), up(button int), move(x, y int, element WebElement), cancel()
Lunch.
<gitbot> [13web-platform-tests] 15Ms2ger pushed 1 new commit to 06master: 02https://github.com/w3c/web-platform-tests/commit/c227d5cd4aa70eb2d638de33518a02c8bc0f68e7
<gitbot> 13web-platform-tests/06master 14c227d5c 15Ms2ger: Remove backup file.
<wilhelm> Scribe: jimevans
<ato> jimevans: Above I posted the situation so far.
simons giving a summary
two tasks remaining
first, how to cancel and reset all user input state
second, how to do a high-level "do what i mean" method
MarcFisher: proposes automatic reset of user state
discussion around how to reset user input state.
MarcFisher: this should be a separate end point on the API
decision reached. send a delete verb to the actions endpoint is a "reset state command"
simons: how do we design using high-level interaction commands?
high-level actions imply clearing of input state before and after action.
wilhelm: suppose i have a sequence where i've left the input state in a certain state. can i stop and inspect the document?
answer is yes, that's what it's designed to do
MarcFisher: json payload should have a sequence id for cross-action sequencing
mdas: you don't need to do that
MarcFisher: simply correlating the arrays element index is error-prone.
discussion of description of json payload for action sequences
LukeInman-Semerau: what if someone sends malformed or invalid JSON for the action?
simons: throws an error in that case
MarcFisher: devices can define what they accept as valid input
keyboard and mouse devices should probably be singleton
tangent on how many fingers are supported by the microsoft surface table product
seva: did we forget about touchpads?
jimevans: for level 1 we are pretending touchpads do not exist.
the API is fully extensible for any other devices.
simons: does the JSON payload for a specific action require a type property?
acceptable types to be defined in this spec: key, mouse, touch
other types can be defined using vendor prefixes.
discussion about how strict the typechecking of attributes in action JSON
specific example: a "tilt" attribute in the JSON object. how do we implement?
additional action names and additional properties in the actions will be handled by vendor prefixes until standardized in later spec levels
<scribe> unknown action types or unknown fields in existing action descriptions will all return an error code.
<scribe> ACTION: mdas to strictly specify all supported device types, action types, and parameters in each action type. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action06]
JohnJansen: can we standardize on pen actions?
simons: once this change has been landed, "pen" input actions can be added
<scribe> ACTION: JohnJansen to patch the spec with pen input actions [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action07]
discussion around which wait times are required for the wait action
<scribe> ACTION: mdas to define symbols for wait times [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action08]
ShuotaoGao: which action should happen first in a multi-action sequence?
as close to simultaneous as possible.
seva: since they can't be actually simultaneous, should we specify the order?
simons: there may be a
millisecond between the actions, but it should be close
enough.
... once we've got an implementation, since no one has
implemented this...
mdas and DavidBurns: we have!
simons: well, not this brand-new implementation, so when that is done, we can see what issues, if any, occur.
JohnJansen: sequence of "simultaneous" actions happen in the order the actions appear in the array.
mdas: we don't control the event queue
simons: no, but you can control what order you put things into the event queue.
MarcFisher: but what if an implementation decides to always process keyboard first, then mouse, for example,
simons: it's still an implementation detail to be left to the discretion of the implementor.
decision: order of actions within the array will not be specified.
<simons> https://www.w3.org/wiki/WebDriver/2014-July-F2F
simons: how do we handle input elements with different (think HTML5) inputs?
discussion from previous f2f, which never got added to the spec.
<scribe> ACTION: MarcFisher to add information to the spec about additional types of input elements. [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action09]
mdas: how will i run the wpt tests that are webdriver tests when i have a driver that uses a tcp socket (not an http transport)?
JamesGraham: the test runner is configurable for different browsers, so we should be able to allow that.
ato: one common api, but separate
code will be required for launching each browser
... the life cycle of the browser should be managed by the
runner, not the tests themselves.
<AutomatedTester> ShuotaoGao: https://github.com/SeleniumHQ/selenium/pull/168
<JohnJansen> scribe: JohnJansen
seva: test runner should manage
browser sessions.
... does it already do that? or need an action?
jgraham: don't need a new action, it's an ongoing effort
<simons> http://www.w3.org/2014/02/25-testing-minutes.html
ato: I'm working on it.
<wilhelm> http://www.w3.org/2014/02/25-testing-minutes.html
<lukeis> http://www.w3.org/2014/02/25-testing-minutes.html#ActionSummary
1. Add definition for getElementText to the appropriate section of spec
simons: already done
2. ato to research root element behaviour in HTML and XML
lukeis: I will create bugs for these
AutomatedTester: bug has been logged against CSSWG
<AutomatedTester> https://www.w3.org/Bugs/Public/show_bug.cgi?id=23825
<simons> MikeSmith is about to get an action item…
<simons> Excellent.
[going over actions from last time and logging bugs]
<simons> http://www.w3.org/2014/02/26-testing-minutes.html#action30
<simons> JohnJansen to investigate setting up a mechanism to determine which requirements in the spec have tests
all Actions from previous F2F have been modified into bugs
as necessary
[break]
simons: sendKeys() to the date format input type
<seva> seva: raised https://www.w3.org/Bugs/Public/show_bug.cgi?id=26290 and 26291; others were "to write tests" which we decided not to raise and hope for bug 26291 to take care of tracking.
jgraham: go through HTML spec element by element
marcfisher: when you sendKeys()
to a field that makes sense to append, we append. With color
(for example), we replace.
... I have an action to update the spec with this language
JimEvans: that answers one question. #2 are we considering using low level actions as the basis for higher level actions.
<selbot2> 03
<selbot2> dawagner@gmail.com closed/Fixed DOCS: Typo in PageFactory documentation - https://code.google.com/p/selenium/issues/detail?id=2 [Type-Defect ]
simons: you should be able to model them, but not required
<wilhelm> http://www.w3.org/2011/08/browser-testing-charter.html
wilhelm: we have a charter to 31
Dec 2015
... milestones, however need to be met:
ato: only AutomatedTester is editing the spec, and he seems too busy to meet the deadlines
jgraham: I think one person owning it makes sense, though, because he doesn't need to wait for others. On the other hand, having that same person write all of the tests doesn't make sense.
it's difficult to know what time I'll have over the coming months
simons: hard to carve out time, but if we schedule meetings, it might work well.
jgraham: CSSWG has done a good job jointly writing specs, but that is the rare example.
ato: unrealistic that David can do this all himself. Perhaps mdas could take some actions from David.
<gitbot> [13web-platform-tests] 15hayatoito closed pull request #899: shadow dom: add code confirm activeElement is readonly and refactor the ... (06master...06submissions/iseki-masaya.activeElement) 02https://github.com/w3c/web-platform-tests/pull/899
jgraham: makes sense to distribute draft material, and then have one editor
marc
[discussion of recurring spec hackathons]
mdas: what are the major parts we need to draft?
AutomatedTester: there aren't a lot of new parts, but we need to rework a lot of the prose (start from scratch in certain sections)
wilhelm: james, can you help write draft patches?
AutomatedTesting: james's focus right now is on the testing, which is where we need him.
Wilhelm: TPAC is in Oct 30 and 31st
<simons> GTAC is 28-29 Oct
<simons> Starwest is 12-17 Oct
http://www.w3.org/2014/11/TPAC/
so if we're going to be Last Call before TPAC, we need to schedule work
ato: as we write tests we will
find issues in the spec
... I've been writing tests and I will take the spearhead on
the test suite
JohnJansen: I will work to see if the CSSWG process will work with our spec
jgraham: fair warning: it is easy to fall into a trap of just testing the spec
I agree with James. we need ot meet W3C REC requirements but ALSO test the spec, the most interetsting bugs will be combinatorial in nature.
wilhelm: will MS have resources to help?
Yes, but not sure I can convert to Python as well
I'll have my intern make them public even if just Jscript versions to help.
jgraham: some cool metaphor about bugs being flies that safely sail through spider webs of boring test cases
wilhelm: lukeis will you have time?
lukeis: likely, yes. Same with
JimEvans
... some time can be carved out.
wilhelm: so we are carving out test time, next the spec authors need to get together to work on the spec itself
i'll put all of our tests up in git by 18 July.
mdas: I'll work on it this week
<wilhelm> Scribe: wilhelm
<gitbot> [13web-platform-tests] 15sideshowbarker closed pull request #325: app-URI spec compliance tests (06master...06submission/martap) 02https://github.com/w3c/web-platform-tests/pull/325
ato: The Python test suite is very unorganized, so it'll take a long while for all of us to go through it together.
JohnJansen: So we should look through it individually?
ato: All the tests require
critical review.
... The spec has also changed dramatically since the tests were
written.
JohnJansen: When we submit tests, we flag them as being not reviewed?
ato: All the current tests must be reviewed.
JohnJansen: I previously assumed they had been reviewed.
ato: We started with the tests in
a hg repo. We have since moved them to wpt.
... Many tests have not been reviewed.
... The majority.
... They even need a re-review now that the spec has
changed.
jgraham: This is a perennial problem with writing tests against a spec that is also changing.
ato: My gut feeling is that we should revisit this topic at the next F2F.
jimevans: That isn't until TPAC, though.
ato: What if I go through them, and send out an email?
jgraham: Yes, it makes more sense to use async communication.
ato: Yes, especially code review.
seva: Is this to check that the group is okay with the tests?
Marc: I changed the tests to use the new driver.
ato: One of the sections I'm most
unsure about is the visibility section.
... 90% of the tests are not covered by the algorithm.
... SHould we remove those, or add them to the algorithm?
Marc: both.
simons: Yes.
seva: Where are the tests from?
ato: Written by me.
seva: I added some of those.
ato: There's also fresh tests that are not in Selenium.
<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py
JohnJansen: The test with omitted body tag, both Chrome and FF fails.
seva: It tests what the spec
says...
... I submitted a fix of this a week or two ago into the
atoms.
... ChromeDriver uses old atoms.
JohnJansen: We'll need to choose if we should comply with the spec, or interop.
lukeis: Compliant.
JohnJansen: What we find is that the web often doesn't agree...
seva: With the body element, we found the root cause. Fixed two weeks ago.
wilhelm: If the tests are more detailed than the spec, the behaviour should go into the spec.
JohnJansen: I could just call out the 14 tests that both Chrome or Firefox fail.
jgraham: Once we have wptrunner running, we can get results for multiple browsers in JSON.
JohnJansen: We are making decisions on these tests...
<JohnJansen> https://github.com/w3c/web-platform-tests/tree/master/webdriver
<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py
<JohnJansen> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py#L190
seva: This issue has been fixed.
ato: This only makes sense for
HTML, right?
... We should probably have one for HTML and one for XML.
jgraham: By the time you get it, it'll be a DOM tree.
(Discussion on the specifics on this particular test.)
MarcFisher_: This test found a
real bug.
... I think it's worth keeping.
seva: We should modify the wording in the spec to not talk about implicit elements?
(Discussion on special behaviour of <body> in HTML vs XML, quirks mode and standards mode.)
<seva> I was wrong, the spec does not actuslly talk about implicit BODY so there's nothing to change in there
ato: Is't this outside the scope of the spec?
JohnJansen: We're trying to define visibility not as defined by CSS, but as seen to a used.
jgraham: The algorithm is using DOM or CSS properties.
<JohnJansen> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py#L131
RESOLUTION: Follow the spec on https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/visibility_test.py#L190
(Discussion on how to detect which element is on top, judging by z-index.)
<scribe> ACTION: JohnJansen to figure out how IE passes https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/res/element-hidden-by-z-index.html [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action10]
<JohnJansen> https://github.com/w3c/web-platform-tests/blob/master/webdriver/navigation/auth_tests.py#L39
JohnJansen: This test fails in Chrome and FF.
lukeis: This is one of the oldest bugs in WebDriver...
ato: Is this in the spec?
... Selenium doesn't support this.
RESOLUTION: We move 401 auth to level 2
<lukeis> #34
<selbot2> 03
<selbot2> simon.m.stewart open/Accepted Support BASIC and Digest HTTP authentication - https://code.google.com/p/selenium/issues/detail?id=34 [Type-Enhancement Priority-Medium Component-WebDriver ]
<ato> https://github.com/w3c/web-platform-tests/blob/master/webdriver/element_state/method_test.py
JohnJansen: The test I'm referring to requests and attribute that doesn't exist. Both FF and Chrome returns an error.
AutomatedTester: They should return null.
MarcFisher__: It's quite possible that our special casing is broken.
(References to different, related tests.)
MarcFisher__: The special casing around class is breaking it somehow.
JohnJansen: The remaining are all the same: six failures because of status code.
simons: Returning strings is correct.
RESOLUTION: Use strings for errors.
(Discussion on compat mode vs legacy mode in the current driver.)
JohnJansen: That's all!
(Discussion of identifier for windows.)
<scribe> ACTION: switchToWindow shall only take window handles [recorded in http://www.w3.org/2014/07/08-testing-minutes.html#action11]
http://www.w3.org/2014/11/TPAC/
Meeting finished!
This is scribe.perl Revision: 1.138 of Date: 2013-04-25 13:59:11 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/Presnet+ AndreasTolfsen// Succeeded: s/Can someone come and collect me at reception?// Succeeded: s/ato: Simon is en route// Succeeded: s/ty// Succeeded: s/:yt never going to give you up// Succeeded: s|Rick Astley - Never Gonna Give You Up - http://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=youtube_gdata|| Succeeded: s|Rick Astley - Never Gonna Give You Up - http://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=youtube_gdata||g Succeeded: s/blacklisted domains/blacklisted domains or protocols/ Succeeded: s/There's/There are/ Succeeded: s/MarchFisher/MarcFisher/ Succeeded: s/AutomatedTester: Escape/mdas: Escape/ Succeeded: s/ACTION: strictly specify all supported device types, action types, and parameters in each action type./ACTION: mdas to strictly specify all supported device types, action types, and parameters in each action type./ Succeeded: s/wait attribute/wait action/ Succeeded: s/itselfr/itself/ Succeeded: s/mot/most/ Found Scribe: ato Inferring ScribeNick: ato Found Scribe: jimevans Inferring ScribeNick: jimevans Found Scribe: JohnJansen Inferring ScribeNick: JohnJansen Found Scribe: wilhelm Inferring ScribeNick: wilhelm Scribes: ato, jimevans, JohnJansen, wilhelm ScribeNicks: ato, jimevans, JohnJansen, wilhelm Present: AndreasTolfsen DavidBurns JohnJansen LukeInman-Semerau MaliniDas ShuotaoGao SimonStewart jgraham jimevans wilhelm MarcFisher Agenda: https://www.w3.org/wiki/WebDriver/2014-July-F2F Got date from IRC log name: 08 Jul 2014 Guessing minutes URL: http://www.w3.org/2014/07/08-testing-minutes.html People with action items: as back conversion cost discuss jgraham johnjansen learning marcfisher mdas of python report results switchtowindow team test to well with WARNING: Input appears to use implicit continuation lines. You may need the "-implicitContinuations" option.[End of scribe.perl diagnostic output]
