Browser Testing and Tools WG, Tuesday, TPAC -- 30 Oct 2012

<sms> Hello everyone

<sms> MikeSmith: yt?

<andreastt> \o

Browser Testing and Tools WG, Tuesday, TPAC

<scribe> Scribe: wilhelm

<MikeSmith> sms, here now

<sms> MikeSmith: we're starting. chat later :)

<MikeSmith> hai

<JohnJansen> I'm going to be in the CSSWG today, but monitoring you guys on IRC.

Screenshots

sms: This section has been renamed from snapshots.
... Summary: Spec mandates screenshots to be transmitted as base64-encoded PNGs.
... (Lossless.)
... In case people want to do pixel-by-pixel comparision.

davidburns: Where would you expect browsers to store this image?

sms: No, this happens on the remote end. Browser just sends it as base64.

andreastt: We support taking screenshot of current window and elements.

sms: Let's look at window first.

andreastt: (Quotes spec on what to take screenshot of.) THis does not sound like taking a screenshot of a window (but of the full document).

davidburns: Yesterday we defined window as top-level browsing context.

andreastt: Window is not visible viewport, but complete viewport.

sms: Assume the user has an infinitely large screen.

plh: Then the name screenshot may be a little misleading.

sms: But snapshot is also confusing.

plh: No, snapshot is not better.

andreastt: Screen capturing is a better word.

sms: Image capturing?

kkania: What's wrong with snapshot?

sms: Not commonly used.

andreastt: Take a step back: our users have requested the ability to take screenshot of full page, screenshot of browser with Chrome, screenshot of the full desktop, screenshot of particular element.

sms: There are tools available to do this for the whole desktop.
... It doesn't feel browser-specific.

davidburns: I don't want desktop screenshots in the spec.
... I can se use case for taking screenshot of browser chrome.
... We could add it, non-normatively.

andreastt: There is no way to take screenshot of visible viewport.

plh: Can't you acess size of what is visible?

sms: Yes, you can determine size of viewport with JS.
... No, we can't take screenshot of arbitrary slices of the page.
... You could take an OS-level screenshot and crop it yourself.
... Different browsers have different capabilites. Chrome would take screenshot of visible part of page. There was resistance against capturing more.

andreastt: We could make it more generic and pass arbitrary coordinates (X0, Y0 to size of window).

sms: Maybe we need to make it less generic: capture viewport, capture element, etc.

plh: If clicking a fragment you expect the page to scroll. A screenshot can be used for verification.

sms: Use scrollheight.
... To determine if an element is displayed, we base it on full page, not just viewport to accomodate different screen sizes.
... There isn't a method to determine if an element is within the viewport.
... You can get the location of the element in the DOM, get window size and scrollheight and do the calculations on your own.
... This is how I would test your use case.
... Does this make sense?

plh: Yes.

sms: Do anyone else than andreastt think we should be able to take the currently visible viewport?

davidburns: I can see value.

andreastt: My ideal solution would be generic solution where you specify coordinates.

kkania: I don't see a good reason for the visible viewport.
... Hesitant to add all these different variants if we can get away with fewer commmands.
... Two commands: full page, specific coordinates.
... Local end could handle the logic.

sms: This leads to an interesting question: If you have a nested frame and switch to that, how would you capture the full size of that?
... "Don't do that"?

davidburns: We could just say you've done nasty things to HTML and you get what you get.

kkania: Lots of pages use iframes and stuff.

sms: Do we do the entire page or just bail?

andreastt: Complicating factors with frames: scrollable frames.

kkania: Same problem with the entire page.

sms: The IEdriver resizes the viewport to the size of the page.
... In Firefox we clone the current DOM into a canvas element.

andreastt: We dump the internal buffer.

sms: That's probably the right way to do it.
... IE/FF method forces a reflow
... Which makes it totally worthless as a debugging aid.
... Do we want to do anything about that?
... Add note to say it's okay to repaint the screen while taking screenshot?

(Agreement.)

andreastt: How would we address an open OS-level context menu?
... Are OS-level dialogs part of screenshots.

davidburns: I'd say no. OS-level things are not part of the screenshot.

andreastt: We have use cases for OS-level screenshots.

sms: This should be an extension to the protocol.

andreastt: Does our window definition exclude OS-level stuff?
... I'd prefer it to be defined explicitly.

sms: We only resize the window once to accomodate inifinite scroll, etc.

jhammel: It could be more explicit for a decaffeinated brain.

kkania: Do we want to include something about windowless and windowed plugins?
... Should snapshot include windowed plugin?

sms: What's a windowed plugin?

kkania: Flash can be a separate OS-level window within the page.
... You may not get Flash content from browser screenshot.

sms: I don't know.
... It would be nice if the screenshot included this content.
... It might be hard to do.

andreastt: It's just blank in Opera.

sms: Should we have a non-normative section on this?
... I prefer if the spec details not only what to do, but why we do it.

andreastt: It might be additional work for the editprs.

sms: It's easy.

andreastt: Note how we expect plugins to behave. We are talking about other OS-level windows here.

plh: Or DRM-protected windows.

<scribe> ACTION: davidburns to write note on plugins and screenshots [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action01]

plh: What do I get when taking screenshot on phone? What zoom level?

sms: iPhone does not reflow on zoom, Android (pre-Chrome) does.

<scribe> ACTION: Figure out what a screenshot should return on mobile (wrt. zoom levels and reflows) [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action02]

sms: There is an empty section on resizing content.

wilhelm: Opera wraps text. Must reflow.

sms: So we should try capturing what the user actually sees.

plh: One way to close the issue is to say you get what you see.

sms: We need to define this clearly. "I'm using assistive technologies" - what happens?
... "Print media" - what happens?
... This should be obvious, but we don't mention it anywhere.

plh: How do I get print shot?

sms: Undefined!

andreastt: I have a few more points for this section.
... We specify body element. That will only work on HTML.
... This won't work on SVG. Or canvas.

sms: We're taking the overall size of the image from the body element.

andreastt: Can we use :root?

(CSS :root element.)

sms: I'm fine with that.

<scribe> ACTION: Use the CSS :root element instead of body element for determining screenshot size [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action03]

http://www.w3.org/wiki/CSS/Selectors/pseudo-classes/:root

sms: This assumes HTML (or SVG) content, but we ought to add a section in the handling-non-HTML-content section on what take screenshot does.

andreastt: What about elements outside the root element? Absolutely positioned at negative position.
... On the Acid3 test, the body element is 1px high, with the content absolutely positioned outside it (but still within the viewport).

sms: The atoms do something fancy here.
... It uses the interactableSize.

andreastt: So that's why it's not working in Opera.

sms: If there is not a scrollbar, we take the window size.
... This is a nightmare.

jhammel: If you have a body that's 400x400 px and your viewport is bigger it would take the size of the viewport?

sms: Yes.
... That is the expected result.
... It is uncommon that the body is smaller than the viewport size.
... (Quotes the interactableSize JS code.)

andreastt: This doesn't work with anything that isn't HTML.

sms: Unless it shoves the content into a HTML container.

andreastt: Is that what we should be doing?

sms: We should worry about SVG content.
... ... We will need to address this at some point.

andreastt: Making the atoms code handle non-HTML content is going to be a lot of work.

sms: Other than SVG, is there non-HTML content we want to take a screenshot of?

andreastt: XML, XHTML.

(Discussion on CSS differences between XML and HTML.)

sms: Does the DOM behave the same way as for HTML?

andreastt: If we rely in CSS here, our lives may be easier.

sms: For XML, we rely on the CSS root element, for HTML we rely on something like interactableSize, which leaves SVG.

andreastt: CSS should work there too.

sms: Are we happy with using the CSS root element?

<scribe> ACTION: Add a section for screenshots for XML and SVG but mark it as non-normative as there are no implementations at the moment. Once we have those implemenations, we need to discuss this again before making it normative. [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action04]

sms: If you take a screenshot of an element, we take a screenshot within the client bounding rect. In SVG this is referred to as bbox.
... Are we happy with the spec on ordinary screenshots, not elements?
... This applies to a top-level browsing context.

Screenshots of elements

andreastt: Is this imeplemented anywhere else?

sms: Not implemented anywhere else (than Opera).

andreastt: This looks like what Opera does.

sms: Except Opera returns a hash of the element.

andreastt: We should change that.
... There is a difference with handling frames, but I'm fine with the described behaviour.

kkania: Does this mean we capture non-visible portions of the frame? I thought we didn't want do to that.

andreastt: I thought this was impossible in IE.

sms: If you are trying to capture the frame, we attempt to move the frame to 0,0 and we capture as much of the frame as we can.
... This would be a bastard to implement in the Firefox driver.
... It might not be, actually.

andreastt: How difficult in Chrome?

kkania: Easy.

sms: I'm happy if browser vendors want to specify the language here. I don't know what's easy and impossible.

andreastt: I can't remember what we do with screenshots of frames.

kkania: Taking snapshot of the entire frame is easy in Chrome.
... Inconsitent with 16.2.

sms: At the begining of 16.3, we should define capability for taking element screenshots. IE would not support this.
... We (the open source project) can not implement this in IE. So the entire feature for taking screenshots of elements will not be supported in that browser. (But will work in other browsers.)

kkania: The functionality in section 16.3 except the final part on frames can be done on the client side.
... What are the use cases for elements?

andreastt: We use it when we have a single-page reftest with two compared elements.

kkania: Real-life use cases outside browser vendors?

sms: It has been a feature request.

jhammel: This has been requested at Mozilla.

sms: If it's only the visible part, you could do all of 16.3 on the local end.
... SHould we break this into two?

kkania: Do we have users for this?

sms: Gmail is in a frame.

(Turns out not to be the case anymore.)

sms: Do we drop 16.3 entirely?

kkania: Not a strong enough use case for the biggest audience.
... Can be easily done locally.

andreastt: In the Opera implementation, the screenwatcher allows you to specify an area on the page relative to the page, not the viewport.
... I'd like to see the section staying.

sms: The problem is the last paragarph with nested frames.
... I suggest we break that out, hide it behind a capability, and then that's cool.
... If you support takesElementScreenshot, the browser will do the hard work. If this is missing, we can do this on the local end. Slightly less efficient.
... Section 16.4 will detail nested frames.

kkania: What would the API look like?

eranm: So implementations need to support capturing the entire page?

sms: Yes.
... 16.2 should only talk about the top-level browsing context.

<scribe> ACTION: Section 16.2 should talk about the top-level browsing context [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action05]

<scribe> ACTION: Section 16.4 should be added to handle nested frames, with a new capability [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action06]

davidburns: There has been a suggestion for browser vendors taking screenshots on a timer as a capability. I would like to get people's opinions on this.
... You could almost get video this way if you wanted.
... My initial thought: no.
... Not the job of a browser to do this.
... Potential for bloat in browser code.

andreastt: Is this something we need to add to the spec?
... Can it be solved with 16.2?
... JUst use regular screenshots?

sms: This is a suggestion from someone working on perf testing. Wants screenshots as part of logging.

andreastt: Do it out of process?

davidburns: Local end could potentially do it - or any other process.
... I wanted to bring this up.
... It has been suggested Chrome is willing to do this.

kkania: I'm not in favour of it.

<andreastt> sms: git@github.com:andreastt/inbox-attack-robot.git

<andreastt> sms: `ant play`

Non-HTML content

andreastt: Should we mention explicitly what to do with screenshots in this section?

sms: No, that's in the screenshots setning.
... There is no requirement for implementations to support non-HTML content.
... What non-HTML content should we be able to control?
... SVG, XML, XHTML, canvas, text/plain?
... canvas in part of HTML.
... If people paint their own UI, there is nothing sane that we can do.
... text/plain?

andreastt: Chrome, Firefox, Opera behave the same the way (in a <pre> container).

sms: IE uses a different control.

andreastt: Defer to HTML spec?

http://www.whatwg.org/specs/web-apps/current-work/multipage/history.html#read-text

sms: Boom! Handled!
... Everyone happy with that?

davidburns: Yes.

sms: XHTML

http://www.whatwg.org/specs/web-apps/current-work/multipage/the-xhtml-syntax.html#the-xhtml-syntax

<scribe> ACTION: If non-document content is being handled by a registered handler, then interaction with it via WebDriver is undefined [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action07]

wilhelm: We should probably handle intents at some point, but not now.

sms: Should we say an XHTML document should be treated as an HTML document wrt. interaction?
... Do we need to resolve this right now?

<scribe> ACTION: Figure out WTF browsers are actually doing with this [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action08]

sms: SVG.

andreastt: Our testers have asked for screenshots on SVG documents.

sms: There has been a request for interacing for SVG.
... There's an assumption that the DOM element represented as WebElement is of element type 1.

<scribe> ACTION: davidburns to chase down someone who actually knows something about SVG to help us with this [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action09]

sms: We want to interact with SVG elements as if they were HTML elements.

andreastt: We need to figure out if the SVG DOM is substantially different from the HTML DOM.

sms: Do we want web driver to go into this?
... Should we solve this in WebDriver 1.0?

andreastt: If we can make some basic approxmiation that works across implementations, I think it's worth it.

sms: I'm fine with that.

jhammel: What currently happens with embedded SVG in HTML?

davidburns: You can get to the SVG, but from there you have to do coordinates. It gets scary.

sms: SVG is getting more popular.

andreastt: SVG parsers have been slow. This is improving now.

davidburns: iOS maps uses SVG heavily.

<JohnJansen> is anyone minuting currently? I cannot tell if my irc is broken or if the group is doing something that is unminuted.

<jhammel> JohnJansen: nope; we're introducing webdriver to a comcast representative

sms: XML
... How do we want to handle XML?
... What should a user be able to do with XML?

davidburns: Find elements.
... Querying, getting text, inner part of elements.

sms: Read only?

davidburns: Yes.

<scribe> ACTION: davidburns, andreastt to come up with a strawman propsal for how we handle XML [recorded in http://www.w3.org/2012/10/30-testing-minutes.html#action10]

jhammel: For Firefox and Chrome, PDF documents are actually HTML.
... Would WebDriver work on that?

sms: No, and not in this version of the spec.
... If your browser treats PDF as HTML, no changes are required. If it doesn't, it's out of scope.

Publishing a new working draft before the end of this week?

All those present vote in favour.

Lunch!

<jhammel> wilhelm++

<jhammel> going to nap for real....i got almost no sleep last night and now its hitting me

<andreastt> kkania: https://github.com/operasoftware/operadriver/pull/66/files

Conformance testing of the WebDriver spec

andreastt: Can we use the open source Python bindings here?

davidburns: Mozilla has no issue with moving Apache 2 software into our tree.
... MPL2 is compatible.

<sms> MikeSmith: question: what are our options for moving the existing OSS webdriver python tests into the w3c conformance tests for the webdriver spec?

sms: This is a good opportunity for to fix historical artifacts in our test suite.
... One thing is reusing the same page a billion times.
... What do we need in order to bootstrap this so we can write tests?

<sms> we're discussing http lib from python or the mozilla one

sms: It should be possible for someone with a clean Python install to run our tests.

andreastt: python -m simpleHttpServer

<andreastt> wilhelm: `python -m SimpleHTTPServer`

davidburns: The mozilla one can be programatically started without delegating to a subprocess.

<andreastt> Will serve CWD on port 8000.

sms: Let's wait until someone complains about the Mozilla HTTPD.
... It's HTTP. You can replace the remote end.

andreastt: Which assertion tool are we looking at here?

davidburns: unittest is a standard library.
... It's old and robust.

andreastt: We need a way to pull in driver dependencies and a test runner.

sms: You would start up the server, you would run the server, possibly passing the URL to the end point.

andreastt: How do we specify this on the server side?

sms: I'm trying to keep everything stupidly simple within this conformance test suite.
... We may need to rewrite the command executor.

davidburns: THe tests at the moment assume the OSS project's libraries are running it.

sms: Let's delay the point of rewriting the command exector.
... Let's write some tests intead.

davidburns: How are we going to lay out the HTML the tests rely on?
... We know we will access them through the web server.
... Each test should rely on its own HTML.

andreastt: We could serve them using file://

sms: That will not work on mobile.
... Some browsers like accessing file:// but are okay with http://.

kkania: Chromedriver is fine with file://(.

<andreastt> wilhelm: don't

sms: Conceptually simpler to create new file for each tests.

andreastt: Resources can be shared between tests.

(Consensus on that HTML should be per test.)

sms: How are we going to store the HTML?
... Worried about numbering of sections.

andreastt: Same repo as the spec?

sms: No, separate repo.

<sms> http://dvcs.w3.org/hg/webdriver-test

<davidburns> http://dvcs.w3.org/hg/webdriver-test

sms: Which test should we write first? .get?

(Yes.)

<sms> "get" is the "hello world" of browser automation

<davidburns> https://github.com/AutomatedTester/W3-BrowserAutomation-Tests

<sms> hi again shepazu

sms: I would prefer a mirror that uses the same dvcs.

davidburns: The hg repo is the canoncal one.

sms: Under webdriver-test I will create a directory called 'navigation'.
... What do we call the first test?
... How do we do the grouping?

andreastt: I'd say no to a class deriving from BaseTest.
... Should these tests be under a particular license?

wilhelm: Yes, the same as other WGs use.

andreastt: I suggest we allow pushing to repo instead of pre-commit reviews.

wilhelm: Agreed.

sms: How should we test section 5.2?

(Live writing of tests.)

<MikeSmith> sms: about your earlier question, you can just push them to the repo, right?

http://www.whatwg.org/specs/web-apps/current-work/status-documentation.html

- DRAFT -

Browser Testing and Tools WG, Tuesday, TPAC

30 Oct 2012

Attendees

Contents