07:50:02 RRSAgent has joined #testing 07:50:02 logging to http://www.w3.org/2012/10/29-testing-irc 07:50:08 jhammel has joined #testing 07:50:51 Lachy has joined #testing 07:52:15 RRSAgent, draft minutes 07:52:15 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html wilhelm 07:52:55 eranm has joined #testing 07:54:03 Meeting: Browser Testing and Tools WG, Monday, TPAC 07:54:07 Chair: wilhelm 07:54:33 kkania has joined #testing 07:54:36 Agenda: http://lists.w3.org/Archives/Public/public-test-infra/2012OctDec/0019.html 07:55:27 abarsto has joined #testing 08:00:30 RRSAgent, make log public 08:00:53 jari has joined #testing 08:00:59 Hi jari 08:01:10 I should change my nick 08:01:21 That's better 08:01:24 hi 08:02:02 JohnJansen_ has joined #TESTING 08:02:12 The room's looking a little bare now 08:02:16 (physical room, that is) 08:03:26 ato has joined #testing 08:06:04 Starting in 5 minutes 08:07:04 shadi has joined #testing 08:07:12 (does it have the countdown music for the last 30 seconds?) 08:07:20 it should! 08:08:02 shepazu has joined #testing 08:08:05 I'll be sure to hum it loudly 08:09:11 Scribe: jhammel 08:09:26 shepazu has joined #testing 08:12:00 RRSAgent, draft minutes 08:12:00 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html wilhelm 08:12:35 wilhelm: is the list of would-be attendents publically available? 08:12:39 plh has joined #testing 08:12:56 jhammel: https://www.w3.org/2002/09/wbs/35125/TPAC2012/registrants 08:13:02 thanks 08:13:07 We've had our 5 minutes 08:15:15 And we're off! 08:15:56 wilhelm: For the first session, what is webdriver? What is the state of the spec? What would the attendents like to get out of webdriver? 08:20:08 introductions: Michael Smith, Andreas Tolfsen, Shangyi Chen, Jeff Hammel, Larry Masinter, Wilhelm Andersen, Simon Stewart, [more] 08:23:36 introductions: Byungjung Kim, John Jansen, Ken Kenia 08:24:20 simonstewart: 3 audiences for driving a browser 08:24:37 simonstewart: 1. people that want to drive a web application end to end 08:25:44 simonstewart: 08:27:33 2nd group of people: browser vendors 08:28:16 andreastt: Opera's made a huge undertaking in converting testing to use the webdriver API 08:28:59 simonstewart: 3rd audience: people that are writing specifications on their own who can't write these in pure JS (mostly hypothetical at the moment) 08:29:34 simonstewart: i've thrown together a quick demo 08:30:23 simonstewart: walk through demo: first create a browser instance; get a particular URL ... 08:30:54 simonstewart: webdriver represents a browser instance, but people are interested in elements on a page; [demos how to get an element] 08:32:13 simonstewart: summary: we're going to google, searching for cheese, then arrowing down for autocomplete 08:34:50 simonstewart: the ability to do testing across browsers with only changing the driver is very powerful with rapid releases from browser vendors 08:35:21 larry masinter: there are no assertions? 08:36:04 simonstewart: no, assertions aren't part of the webdriver API; this is left to whoever is writing the tests/scripts 08:36:58 simonstewart: each of one of several languages that selenium has language bindings have their own mechanisms for running tests and doing assertions 08:38:12 simonstewart: there is a JS binding for webdriver for nodejs 08:38:47 simonstewart: there are multiple implementations of webdriver; the OSS project is where the webdriver work began 08:39:25 simonstewart: opera were the first implementators of webdriver; written separately, but using the OSS project to ensure compatability 08:40:11 simonstewart: Opera owns the Opera driver; Chrome is maintaining the Chrome driver; Mozilla is taking responsibility for the Firefox driver with Marionette 08:40:35 Larry Masinter: You want to test scrubbing through a video; is that doable? 08:41:03 simonstewart: there is a section on dealing with non-HTML content; we currently don't have a great story on how to test 08:41:21 simonstewart: example: canvas; writing to canvas isn't generally done with testability in mind 08:42:03 plh has joined #testing 08:42:05 simonstewart: what we want to do with the webdriver API is to bring focus to how to make what is done on the web testable/automatable (social engineering) 08:42:31 wilhelm: isn't the video element just shadow DOM? 08:42:45 simonstewart: one of our topics is dealing with the shadow DOM 08:43:05 simonstewart: other open questions: SVG, a11y 08:43:20 simonstewart: a surprising amount of web pages are still JS and HTML 08:43:42 Larry Masinter: having a roadmap of what needs to be done would be really useful 08:44:01 simonstewart: there is a section on extending the protocol; extending the APIs + vendor extensions 08:44:23 simonstewart: example: wouldn't it be nice to have a API for bookmarks? However this is done differently across browsers 08:44:38 simonstewart: process: experiment; see what works and doesn't; consolidate on what works 08:45:33 simonstewart: just the point of getting a standard that works with JS and HTML and works with the various browsers is a huge undertaking 08:45:46 simonstewart: then figuring out what people really want 08:46:25 Larry Masinter: as a separate item, in the long run, what needs to get tested? Mapping out what all should be tested 08:47:32 Larry Masinter: e.g. performance testing doesn't fit in this framework; but there should be some place where performance testing can be consider (wrt w3c WGs) 08:48:05 Andreas Tolfsen: we can look at the webdriver API as part of that infrastructure 08:48:33 Andreas: we use webdriver to run the tests; then we use another framework actually handle the testing 08:48:37 MikeSmith has joined #testing 08:49:19 Andreas: webdriver needs to be much more generic than just browser testing 08:50:20 RRSAgent, make minutes 08:50:20 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html MikeSmith 08:50:32 Andreas: the screenshot story is a bit complicated 08:51:51 Andreas: if we take screenshots externally, should that be part of the spec? otherwise, this could trigger a window repaint 08:52:48 jhammel: so webdriver is fundamentally a browser automation framework, not a testing framework 08:52:54 simonstewart: yes 08:53:15 Larry Masinter: but it is the browser testing charter? 08:53:45 RRSAgent, make minutes 08:53:45 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html MikeSmith 08:53:53 Simon Stewart: yes, we need to ensure that webdriver supports all that is needed to test 08:54:11 Larry Masinter: I wonder if we could do fuzzing 08:54:37 Andreas: it would be difficult to try to put all security testing into webdriver 08:55:15 Simon Stewart: tests that involve using a browser like a user are good candidates for webdriver automation 08:55:41 Wilhelm: Next topic: State of the union: where is the spec right now? 08:56:14 Simon: I've landed a bunch of edits; the specification is moving forward 08:56:41 Simon: the OSS selenium project; 15-100 checkins / week ; most of this is refining capabilities 08:57:05 Simon: (handling edge cases, etc); the broad strokes are fairly well defined 08:57:56 Simon: however, getting the test cases into the w3c test suite and ensuring that the spec is rigorously defined in English is still needing 08:58:09 Simon: some sections are fairly flushed out; some are skeletal 08:58:36 Simon: the implementations are further along than the specification because of a common test suite 08:58:44 Wilhelm: which sections are finished? 08:58:46 glenn has joined #testing 08:59:30 JonathanJ1 has joined #testing 08:59:51 Simon: Commands+Responses flushed out ...; Switching windows; Running without focus : spec says you should; ... 09:00:18 Simon: if you can use the OSS tests section 10 makes a lot of sense 09:01:13 Simon: cookies = WIP; ...; Modal dialogues: skeletal; screenshots aren't flushed out 09:01:56 Simon: How we do logging in webdriver: logging is out of process 09:02:28 Simon: selenium grid: driver + client may be running on different machines 09:02:50 simonstewart: vs in process; maybe console.log has all we care about 09:03:01 Larry Masinter: You have logging but no assertions 09:03:30 Simon: Yes; we want to get logs back in a consistent format; the logging API we have lets you get consistent records 09:03:44 Larry Masinter: The logs would include assertion failures 09:03:55 Simon: No; it is only internal logging 09:04:12 Larry Masinter: What do you need to do with the logs that require standardization 09:04:20 e.g. the console of the browser 09:04:33 Larry Masinter: If different browsers log differently, does that matter? 09:04:48 Simon: There isn't a standard for the content of the logs 09:05:52 Simon Stewart: you're probably on a heterogenous network of machines; if a test fails, the user doesn't have any insight into what is going on 09:06:33 Simon Stewart: developers want to take a look at the logs at all intermediate stages; this is what happend locally, on a webdriver server, on the test machine, etc 09:06:49 Simon: we need that format in order to be able to get the logs 09:07:20 Simon: we haven't put assertions in, but we've given you the ability to put assertions in 09:07:32 Simon: its not active logging; its fairly passive 09:08:55 Larry Masinter: roadmap: how do you ensure adequate consistency? are we writing our specs too precisely? 09:09:58 Simon Stewart: i've seen a number of projects that have survived major reworking of the UIs without failing; the key is how you find the elements 09:10:45 Larry Masinter: there may be operations that can only be done with multitouch, etc 09:11:09 Simon Stewart: you can query your webdriver instance for the capabilities it supports 09:11:24 Simon Stewart: e.g. do i support touch actions? 09:11:53 Larry Masinter: would you use media queries to determine what tests are run? 09:12:16 Simon Stewart: no; in the OSS project, you only run tests if the assumption is true 09:13:23 Simon Stewart: we've divorced ourselves from a testing framework, but we've given you the capabilities to create a testing framework; we allow you to query the capabilities without enforcing you to do anything if the capabilities aren't met 09:14:23 John Jansen: How do i easily determine the diff between red+blue? 09:14:49 Simon Stewart: using diff from the VCS; would you prefer a different way? 09:14:55 byungjung has joined #testing 09:15:20 John: is that pushed decided by the wg? 09:15:28 Simon Stewart: Yes 09:15:53 Wilhelm: is there any particular reason why you care about the blue one over the red one? 09:16:01 John: mostly out of habit 09:16:15 Simon: because of the way we're writing the specification, blue isn't invalid 09:16:40 http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html#commands-and-responses 09:16:41 Wilhelm: Blue refers to the published working draft; red is the editor draft 09:16:50 http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html 09:16:55 http://www.w3.org/TR/webdriver/ 09:17:50 jhammel: those links probably won't be in the minutes unless you scribe them 09:18:00 Wilhelm: What do you all want from the webdriver spec? What do you want from this meeting 09:18:26 location of webdriver spec: http://www.w3.org/TR/webdriver/ 09:18:32 thanks :) 09:18:46 andreastt: we want web developers to be able to use opera when testing 09:18:48 darobin has joined #testing 09:19:03 Andreas: to ensure that they have site compatability 09:19:36 Andreas: last year, we released our HTML5 parser; we also got sites to run our test suite versus their sites 09:20:41 Jeff Hammel: trying to figure out what Mozilla needs to do to get Marionette + webdriver on spec 09:21:17 Shangyi Chen: trying to observe what is the progress is being done 09:21:45 Shangyi Chen: Baidu using webdriver 09:23:05 John Jansen: trying to solve big problems at microsoft; this is one point of input 09:23:27 Wilhelm: last time i wanted to test my browser; today i want to test my sites with various browsers 09:23:50 Wilhelm: want the spec to move forward; want all WGs to test all the specs they're working on 09:24:18 wilhelm: want to be able to point to a spec to say, 'Use this spec to test all other specs' 09:24:40 Simon: working on webdriver for 7 years now; has become the de facto way of testing web apps 09:25:07 Simon: I want to standardize it; as the web gets more capability, it becomes harder and harder 09:25:17 abarsto has joined #testing 09:25:26 Simon: need support from the browser vendors; webdriver is one of the tools in the arsenal 09:26:00 Simon: the thing that testers hate doing is duplicating effort; they tend not to use another API once they have one solution working 09:26:59 ?: Make sure the spec is understood; the more help we can get from browser vendors, the better 09:27:28 ?: figure out the limits of the spec; figure out what can be done to extend these limits 09:27:45 Ken Kenia: primary concern for chrome team is website compatability 09:28:00 Ken Kenia: can test websites across browsers quickly 09:28:21 Ken Kenia: we don't use webdriver as pervasively as Opera 09:28:31 Larry Masinter: Does webkit use webdriver? 09:28:40 Ken Kenia: not currently with internal testing 09:28:48 Larry Masinter: should it? 09:29:20 Simon: there are at least two projects that are based on webkit that need to do testing; so yes 09:29:41 Simon: it simplifies the vendors work; it simplifies the driver's work 09:29:59 Wilhelm: is there anything this WG can do to help with the politics of webkit? 09:30:11 Wilheml: if so we should do it 09:30:38 Simon: webkit is worked largely on by Chromium; though Apple depends on it too 09:31:21 Simon: is there anything we can do to encourage participation by MS? 09:31:30 John: let's talk about that later 09:32:15 sms has joined #testing 09:58:15 RRSAgent, draft minutes 09:58:15 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html wilhelm 09:59:44 JohnJansen has joined #testing 10:03:54 davidburns has joined #testing 10:04:01 kkania has joined #testing 10:04:21 eranm has joined #testing 10:04:27 Here we go again! 10:04:33 Chair: sms 10:04:44 Chair: simonstewart 10:04:48 Scribe: wilhelm 10:04:49 Ohhh... I'm back as sms 10:04:54 Interesting 10:05:19 Topic: Discussion of open issues in the spec 10:05:27 http://lists.w3.org/Archives/Public/public-test-infra/2012OctDec/0019.html 10:05:31 byungjung has joined #testing 10:06:01 (See list of open issues in the agenda.) 10:06:49 eranm has joined #testing 10:07:14 sms: Some parts of the spec is integer-heavy. (Points to error codes.) We could use strings instead. 10:07:31 Topic: Interoperability 10:08:00 sms: The spec, as written, does not describe a transport mechanism and how to encode data. 10:08:27 JonathanJ has joined #testing 10:08:46 sms: This is for historical reasons. Different drivers use different protocols. 10:09:38 sms: It is possible to follow the spec - and not be interoperable. 10:11:03 sms: Suggestion, listed as a note in the spec: implementations should allow the use of JSON wire protocol. 10:11:49 sms: It would be possible to write an implementation specific to one language. For example a proprietary connector for mobile. 10:12:29 sms: How do we allow people the freedom to choose the transport mechanism they prefer? JSON+HTTP may not be the most efficient. 10:13:02 sms: At the same time introducing a new device/platform/browser must be easy. 10:13:09 jhammel: The goal is to encourage it, not enforce it? 10:13:16 sms: We could say "must". 10:13:33 rrsagent, draft minutes 10:13:33 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html JonathanJ 10:13:34 sms: This is currently in a non-normative section. 10:13:59 sms: We could make that spec normative. 10:14:18 wilhelm: Why shouldn't we? 10:14:36 sms: Nokia was against using JSON over HTTP. Maybe they don't mind a shim. 10:15:07 sms: One suggestion is we mandate that every implementation must have the shim. 10:16:46 (Tangent about malicious commits to the conformance test suite using something else than the JSON wire protocol.) 10:17:30 andreastt: Has other specs faced this problem? 10:17:44 sms: WebDriver is the only out of process spec that tries to have a consistent API. 10:18:20 andreastt: It makes sense for me to enforce this for the spec to be practically useful. It should support a unified transport protocol. 10:18:26 sms: This is a fairly safe tech stack. 10:19:30 ACTION: Make the section on the JSON wire protocol normative 10:20:04 (Discussion on whether all drivers will be part of the open source project.) 10:22:00 (Discussion on language agnosticism.) 10:23:32 sms: If you go into a Microsoft shop and say "I want to use this Java program", that's not going to work. By having a standalone executable people can choose the bindings they prefer. 10:25:37 sms: I don't want a duplication of effort in the transport layer. If you use the JSON wire protocol, and have a shim, you can use any langauge or driver. 10:27:00 sms: The shim should be on the side of the browser vendor. 10:27:44 kkania: Is there a reason for out-of-process instead of in-process executable? 10:27:57 sms: This is a may. Implementors are free to bake this into their browser. 10:28:37 andreastt: These are implementation specifics, no? If you are writing something for your own uses, this doesn't cause any trouble. 10:28:53 kkania: Is there a reason for why we want JSON over HTTP? 10:29:20 sms: It works remotely. It works over every single client language you can think about implementing. Even more than loading a DLL. 10:29:34 sms: Works independently of bitness (32 vs 64 bit). 10:29:50 andreastt: And this only covers the C implementations. 10:30:33 sms: We settled on JSON over HTTP because all these languages have a HTTP and JSON library. We considered thrift and protbufs. 10:31:01 sms: I wrote a JS client that used XHR. 10:31:17 darobin has joined #testing 10:31:27 sms: One of the major audiences are people testing their web applications. Many of these people don't have root. 10:31:44 sms: They might not even allow you to install any software. This rapidly became a nightmare. 10:32:03 andreastt: In the millitary, you may not even have access to the Internet. 10:32:22 jhammel: You might even do this manually via telnet, typing everything. 10:33:15 sms: This is currently covered by appendix E. This will be moved into the body of the spec, as normative prose. 10:33:24 Topic: Internationalised input 10:33:55 sms: There are two use cases for internationalised input. 10:34:18 sms: Am I doing proper roundtripping with international characters? 10:34:40 sms: I want to make sure my app does the right thing as the user is typing using an IME (or similar). 10:35:30 sms: If you use sendKeys with international characters the character makes it into the input field, but it is nothing like what the user actually does. 10:35:58 eranm: IME is only used for complex alphabets. 10:36:23 andreastt: We have not done any work on IMEs. We don't check if the keys exist on the keyboard. 10:36:52 andreastt: There have been several requests from our users for support of changing the charset of the keyboard. This is a difficult problem. 10:36:57 sms: Mobiles have different input methods. 10:37:20 sms: Choose a different keyboard, long press on E to get É, etc. 10:37:42 andreastt: On fature phones you have predefined buttons that can be programmed. This also applies to some smartphones. Programmable keys. 10:37:57 andreastt: You don't know what's going to be on the keyboard. 10:38:31 sms: The IME stuff we have is very desktop specific and requires a lot of knowledge about the machine you're running on. Is this a reasonable thing to have in the spec? 10:38:55 eranm: Since we've added support, nobody has actually used it. 10:39:15 eranm: There are 1.5 billion people using complex scripts. Solicit input from them? 10:39:29 eranm: Hebrew/Arabic are much simpler. What we have may be good enough. 10:40:03 sms: So we should move IME to non-normative. 10:40:31 andreastt: É is registered as two different characters in the browser. 10:40:34 rrsagent, draft minutes 10:40:34 I have made the request to generate http://www.w3.org/2012/10/29-testing-minutes.html JonathanJ 10:40:51 andreastt: Complicating factor: OS specific stuff. 10:40:51 plh has joined #testing 10:41:04 sms: And different keyboard layouts: Norwegian/English/etc. 10:41:18 sms: Sometimes there will be a mapping to a key, sometimes it won't. 10:41:30 sms: If there is a character on the keyboard, we can just send the right character. 10:42:12 sms: In the case where a key is not on the keyboard (shalom in Hebrew) we just stuff in the right characters. 10:42:47 davidburns: We should not go in detail on this. 10:43:11 andreastt: What should the spec cover? I don't think this fits. A library on top of the sendKeys implementation makes sense. 10:43:49 davidburns: This may be covered in the Widgets spec. 10:44:07 sms: I'm hearing: We specificy that you must be able to do internationalised input, but don't specifiy how that is done. 10:44:12 btw, somewhat related to this topic, Norbert Lindenberg of http://norbertlindenberg.com/2012/10/ecmascript-internationalization-api/index.html is here at TPAC this week 10:44:44 eranm: We emulate shift-a for A. Should we spec this? 10:44:48 kkania has joined #testing 10:44:54 sms: We also do the alt keys on Window. 10:44:59 s/Window/Windows 10:46:02 JonathanJ: IME is very complicated. 10:46:40 ACTION: JonathanJ to give input on non-latin character input 10:47:39 ACTION: davidburns to look into how the DOM spec handles keyboard events and internationalisation 10:48:29 andreastt: The current IME section is... nothing. 10:48:56 sms: There is a series of commands for handling IME in code. Document these as non-normative. 10:49:38 eranm: Two ways of using IME: Enable it in the OS, input latin characters. Or give me a list of IMEs and interact with them. We should document both, but the first is definitely non-normative. 10:49:52 eranm: We should listen to people who actually use this. 10:50:22 Topic: ARIA locators? 10:50:56 sms: An accessible web is a good web. Should we add ARIA as another type of selector? 10:51:16 sms: I don't know if the ARIA specs have any APIs that browsers should implement. 10:51:39 sms: Is there an equivalent to querySelector for ARIA? I don't think there is. 10:51:47 JohnJansen: I haven't seen one. 10:52:26 sms: Given the way we implement XPath, CSS selectors, etc - by delegating to the browser - and there is no API for this in the browser, let's not add this. 10:52:33 Topic: Shadow DOM 10:52:36 sms: This'll be fun. 10:52:45 sms: Is everyone familiar with what the shadow DOM is? 10:53:33 davidburns: Shadow DOM is an object model that is hidden away from view. In a