See also: IRC log
<wilhelm> Scribe: simonstewart
I like cheese
wilhelm: starts the meeting
<wilhelm> guest: We're getting you set up now. Starting with introductions.
wilhelm: works as a independent consultant, chair of the browser tools and testing group
eranm: works at google on simonstewart's team. Works on user interactions
jarib: works on webdriver. works at finn.no. Does the ruby bindings
danielwh: works at google on webdriver
I do the same thing
jimevans: works at salesforce.com. Works on Webdriver (the IE and .net bindings)
freynaud: works at ebay on automation and Grid
jleyba: googled. Focuses on JS testing and does the JS bindings
andreastt: works at opera.
represents their browser automation team
raga: works at experia as a test lead
samit: works for TomTom as a tech lead, and the maintainer of the selenium IDE
wilhelm: why are we here, how the
... there are several reasons for writing the webdriver spec.
... wore the test manager hat at opera for several years
... Goal is still the same. The W3 should be able to write tests for the new specs
... The easy stuff has already been done (e.g.: rendering (!))
... Things that use JS assertions are already done too
... difficult part: testing things that require some sort of user interaction
... opera originally considered watir and webdriver. Didn't care about automating websites, just the browser
... in the testing IG at the w3c we're trying to reach a consensus about which tools should be used.
... Goal is to get all w3c standards to be able to use the webdriver api to write tests
... therefore we need a spec
... in addition, there's the benefits for the OSS project to have a proper and clear spec.
... Also, I believe we can use the spec as a tool to bring other people on board with the browser automation effort
... finally, the w3c has a patent policy that members of a working group must follow.
... reduces legal risk for people implementing the standard
... it's also my birthday.
... we'll talk until lunch and then clear up and disputes in the spec afterwards
raga: will this replace any other specs?
wilhelm: will just focus on
browser automation, allowing others to test the things that
... notice that this is an informal meeting, so anyone can join in. Would be happy to help new members come and join the group
... I hereby invite all of you as invited experts to the WG.
<wilhelm> Scribe: wilhelm
simonstewart: I've never done
this before – if you have questions, just ask.
... Skipping the abstract…
... Leaping straight into the introduction.
… Missing expected use cases in the introduction.
… Three use cases should be added: people who want to test web sites, …
… people who are working on a browser itself and want to verify that the browser works correctly, people who write specifications that define how a browser should behave.
… We would be dead in the water if we were to define everything independenlty. We cross-reference other specs where applicable.
… If you find an error in the spec where we should be referring to another spec, please let us know.
… Naming the two sides of the API. Traditionally you define client and server. These are overloaded terms.
… One end is the test or the user, one end is the browser that is being automated.
… Deliberately not mentioning how things are routed. How commands are formatted are up to implementors.
… There will be a non-normative section on the JSON wire protocol, which the most common implementation.
… We intend to provide as much as possible in Web IDL format.
… The webdriver spec will lay out commands that can be used, as well as a way these can be extended by vendors, avoiding clashes.
… (Detailing steps of commands and responses – see spec for exact details.)
… We'd like people to be able to do send additonal information with commands and responses. (Screenshots on error, authentication, &c.)
… Open question: should we require implementors to preserve the order of the fields?
others: Not important.
simonstewart: The numeric error
codes have gaps due to legacy.
... … There's definitely a hierarchy of error codes, similar to in HTTP.
… Yes, perhaps we should group them together.
jleyba: We have implicit error codes as well due to the underlying HTTP.
jimevans: We're not specifiying that the transport will be HTTP.
jleyba: We should validate commands. If you don't give the expected parameters, server may blow up.
simonstewart: Missing part
... Status codes should encapsulate all possible errors.
jimevans: Currently we don't.
jleyba: We should have classes of error codes where vendors may extend (say 600 range).
simonstewart: Users generally don't do the right thing with error codes.
kevin: Are integers defined to be unsigned in WebIDL?
simonstewart: These are unsigned.
rambighananthan: When our scripts look for elements, they may not be in the right place. Is there a way to assert whether an element is within a given area?
simonstewart: This is already possible on the client using the available information about position and size.
… Goal is to keep the spec small. We want to define the building blocks people can build more complex things with.
<scribe> ACTION: Go through the status codes [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action01]
simonstewart: Spec uses capabilities in two different ways. Desired capabilities. Capabilities object for responses – this is what I support.
… The interface has a map of a dictionary of capabilities.
… Occasional objects.
… Two methods: boolean has method, get method.
kevin: In section 3.1.1 doesn't allow arbitrary objects to be returned.
simonstewart: How do you encode objects to send across the wire?
… Maybe we should tighten up the spec here.
andreastt: The capabilities we have now is an unholy mess.
kevin: How is null represented in Web IDL?
simonstewart: I don't know Web IDL very well. Feel free to correct things.
samit: Are we planning to standardize capability names?
simonstewart: We will standardize a set.
… We need a non-normative appendix of capability names with a link back to the section that defines it.
simonstewart: Some vendors may
use other protocol formats.
... Section 3.1.1 – must be of type boolan, string, numerical, dictionary.
… Dictionary can contain keys of type string and types above.
… Serialization format must be clearly defined. Simplest is dictionaries.
… This section is merely defining data types.
kevin: It should be called out that the absence of a value is NULL.
jleyba: get returns the raw value of a capability and makes an interpretation of that value.
simonstewart: Many questions here are due to me not understanding how to write Web IDL. Text may be easier to read.
<jari> (union types: http://dev.w3.org/2006/webapi/WebIDL/#idl-union)
samit: Can you give an example of when mutable capabilities are needed?
simonstewart: “I want Firefox 3.6 with this particular profile”.
… You might not know this when you start.
dawagner: Is this an implementation detail?
simonstewart: I'm happy to drop this.
jleyba: I vote for dropping it.
simonstewart: I'm fine with that.
<scribe> Scribe: jari
simonstewart: topic: sessions
simonstewart: each session given uuid, opaque string
discussion removing the requirement of a uuid
simonstewart: (describing sessions - see spec)
jleyba: do we want to define a format for the command name?
simonstewart: want a format that allows for vendor extension
jleyba: current names based on firefox implementation, should we standardize on that?
simonstewart: nice to have human
readable command names
... do we want command names to be case sensitive?
eranm: no benefit
<scribe> ACTION: section 3 - specify command names as case sensitive [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action02]
simonstewart: (continues reading from the section 4.1)
SamitBadle: change sessionId from SHOULD to MUST?
<SamitBadle> not, just asking.
simonstewart: why is explained later in same section
jleyba: asking if the remote end
needs to respect sessionId created by the local end
... some capabilities should be set in stone
simonstewart: client should check what it gets back
discussion around fuzzy capability matching
<scribe> ACTION: Add a "required capabilities" parameter when creating the session (Section 4.1) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action03]
<guest> pmfji for 'required capabilities' you might define an 'isRequired(capability)' method.
discussing problems around required parameters when the capabilities object is an unordered set - the first capability checked would fail
jleyba: spec should say the remote end should collect as many capabilities as it can before it fail
simonstewart: i'm ok with that
discussing required capabilities in the context of grid implementation (i.e. intermediate nodes)
simonstewart: (continues reading from section 4.1)
eranm: what was conclusion on UUID?
simonstewart: remote end free to ignore UUIDs provided by the client
jleyba: don't see why local end should ever be able to set the UUID
simonstewart: known use cases exist, do we want to close the door on those?
<scribe> ACTION: server should make sure the UUID provided by the client is unique, not used before (Section 4.1) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action04]
<scribe> ACTION: fix "0" string in 4.1. point 6 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action05]
andreastt: how to specify different products in the capabilities?
simonstewart: use browserName
jleyba: are we standardizing browserNames?
simonstewart: no. ultimately each
implementation will pick a browserName
... (reading section 184.108.40.206)
<guest> pmfji: perhaps use a dotted notation for capability names, e.g. "platform.version" instead of platformVersion etc.
simonstewart: just taking what's already there
SamitBadle: could we leave out "ANY" and just not send a platform capability?
kmenard: spec says "may be used"
simonstewart: may also be used to indicate that current OS is unknown from the remote end
hugs: what capability would differentiate e.g. iPhone from iPad?
simonstewart: spec says implementors may add additional platform names
(reading through section 4.1.2)
simonstewart: missing from this
error handling section, required capabilitiy not
... (reading section 4.1.3)
discussing the non-normativity of this section
simonstewart: (reading from Section 5)
<scribe> ACTION: Navigating between HTTP vs HTTPS domains [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action06]
andreastt: are we restricting ourselves to dealing with HTML?
simonstewart: opens up a can of worms dealing with other things
wilhelm: needs to deal with XML content
simonstewart: page load
strategies deals with HTML/HTTP
... can have non-normative section on XML
jleyba: is the intent that these strategies apply only when explicitly say they apply, or do they apply to every command?
simonstewart: see section 5.2
<scribe> ACTION: link to HTML spec for document.readyState [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action07]
<scribe> ACTION: Say explicitly that vendors can add additional page load strategies [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action08]
<scribe> ACTION: Link Section 5 to Section 13 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action09]
simonstewart: (reading section
... (reading section 5.1.1)
<eranm> ACTION: Clarify in section 5.1.1 that this applies to all kinds of invalid certificates, not only self-signed ones. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action10]
<scribe> ACTION: Section 5.1.1 shouldn't use WebDriverException [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action11]
simonstewart: page load strategy specifies when the next command will execute
<scribe> ACTION: Swap order of 5.1 and 5.2 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action12]
breaking for lunch
<eranm> scribe: eranm
wilhelm: morning session was useful, suggest continue in the same format - going over the chapters
simonstewart: browser tabs are counted as separate windows
jleyba: window.top is a frame, happens to be the top-most frame. Say that when switching to a window you switch to a top-most frame.
wilhelm: the html specification uses the term "browsing context"
simonstewart: todo: define
... 6.2 window handles.
simonstewart: suggesting a window class to refer to
jimevans: window IDs returned from the IE driver are no longer the HWND, this is an opaque string
<scribe> ACTION: we should make clear that the window handle is unique within the session [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action13]
andreastt: assuming we're talking about visible windows
jleyba: chrome also has hidden windows, we may have to be more restrictive
<scribe> ACTION: andreastt and jleyba, re-word the language in 6.3 on the windows returned from there [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action14]
jimevans: suggests specifying that the order of handles returned from getWindowHandles does not imply anything about the order of opened windows
simonstewart: the returned value will be changed to a set of strings to emphasize the order is not defined
freynaud: calling window.close is difficult for the grid, because it does not know the driver is done when the last window is closed
<scribe> ACTION: in section 6.4, take out the equivalence to quit. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action15]
use case brought up by SamitBadle is the last window of a mobile browser
(use case brought up by andreastt )
<scribe> ACTION: maximise should be in 6.5 as well [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action16]
andreastt: what happens when the command is called for a browser that cannot be resized?
<scribe> ACTION: define an exception if the browser cannot be resized to the specified size [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action17]
(case: chrome cannot be resized to less than 200 pixels)
<scribe> ACTION: maximise and full screen are needed as commands. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action18]
jimevans: maximising and resizing to the screen size are not the same
simonstewart: leave positioning
out for now, not enough use cases to justify it
... explaining os-specific scaling (window zoom).
jleyba: is that like pinch gesture on mobile? others: no
<scribe> ACTION: jleyba and andreastt will write section 6.6 about window zoom [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action19]
jleyba: what happens if the window is no longer there? simonstewart : an exception is thrown if the window does not exist anymore
eranm: when a new window opens, we should specify commands are still processed in the original window. simonstewart : handled in section 7.
<scribe> ACTION: fix typo in 7.1, window._top -> window.top [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action20]
jleyba: section 7.2, if the id attribute is not standard on window, we should not check for it
wilhelm: editorial comment: use normative words in section 7.2, (must/should)
<scribe> ACTION: use normative words in section 7.2, (must/should) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action21]
<scribe> ACTION: jleyba : document which commands are safe to execute without a window. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action22]
simonstewart: closing the last window is not possible only on some platforms (mobile), definitely possible on others (OS X)
eranm: does the section about multiple windows in 7.2 only about the multiple windows being opened on start-up?
simonstewart: could also happen
when attaching to an existing browser
... when you create a new session, if that new session creates only one window, this is the current window.
... in case more than one window is being opened, then you have to make a choice
<scribe> ACTION: change throws clauses to error clauses (errors are returned, could be translated to exceptions in the client side) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action23]
<scribe> ACTION: add null as an option in 7.3 (frame specification can be null, in which case it's the top-most one) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action24]
<scribe> ACTION: be explicit in 7.3 that domain boundaries can be crossed [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action25]
<scribe> ACTION: section 7.3, step 2.1 should be changed to null (see action 24) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action26]
simonstewart: section 7.3, step 3: so frame/iframe elements could be found and used in a switchToFrame command
jleyba: do we want the ability to switch up? simonstewart: users can find the way down again
SamitBadle: what happens when switching to a frame then switching to another window and switching back? simonstewart: specification says switching to a window switches to the top-most frame in it.
jleyba: suggest specifying that all IDs, unless specified otherwise, are UUIDs.
<scribe> ACTION: specify that all IDs, unless specified otherwise, are UUIDs. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action27]
klm: introducing himself: Michael Klepnikov, working at Google on latency testing, interested in WebDriver. Specifically interested in browsers exporting existing information about timing of actions.
switching to section D, logging:
simonstewart: the basic idea is
that there's a logs interface which is queried for a specific
kind of logs: browser logs, etc.
... this is a uniform interface for getting logs for any stage, from the test itself , through the grid, to the browser launching itself
wilhelm: which problem are we trying to solve?
simonstewart: the test has failed, user would like to see the reason of the failure: out-of-memory in an intermediate step, commands executed before finding an event timed out.
<simonstewart> davidburns: do you want to be dialed into this meeting?
<davidburns> yea for a bit if that is ok?
<davidburns> simonstewart: ^
<simonstewart> eranm or dawagner: do you know the incantation to get davidburns in?
dawagner, can you invite?
<davidburns> last time we did a Google hangout
<dawagner> Yep, davidburns: Email address?
simonstewart: define a mechanism for moving logs
davidburns, is that your google+ account?
<davidburns> eranm: yup
<dawagner> You should have email
simonstewart: when calling getlogs(), do I get the partial logs since the last invocation or all logs ?
wilhelm: mentions another spec about logging.
SamitBadle: which performance data are we interested in?
klm: will instruct the browser to start collecting the performance data. Once that's collected, it should be in a separate channel, not polluted with other logging data
<davidburns> simonstewart: fyi https://twitter.com/#!/AutomatedTester/status/192764437348880384
freynaud: provide incremental logs, so you know which period of logs you're getting
klm: don't see a difference between console logs and timing logs
simonstewart: suggest passing a
filter when fetching the messages
... start recording X log type, stop recording X log type, get all categories of log types, clear logs.
klm: if there are two parts of the test framework that don't know about each other, incremental vs. full could be a problem
simonstewart: clear is called for a specific type, subsequent get will return logs from the last clear or start record commands
andreastt: some logs are document-specific, some logs are server-global, do we mix them?
simonstewart: call clear and get logs for logs you care about.
simonstewart, andreastt : when a session is deleted, the logs should be deleted as well.
simonstewart, the spec should define how to clear and collect the logs, each log type will have specific details which will not be specified here.
<scribe> ACTION: simonstewart : write section D on logs [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action28]
simonstewart: the spec does not
define the protocols or the transport layer, how commands are
... different types of encodings are suitable for different kinds of devices.
<scribe> ACTION: eranm: section 9.1, the WebElementList SHOULD be immutable. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action29]
<scribe> ACTION: simonstewart switch WebElementList to array of WebElement (native WebIDL) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action30]
jari: there's even a readonly type.
hugs, can we move XPath to be last? simonstewart : other locator strategies may be implemented using that. Agrees to moving it down.
<scribe> ACTION: jleyba section 9.2.3 shorten capability name [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action32]
<scribe> ACTION: jleyba , section 9.2.4 re-phrase to use CSS selectors other than xpath. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action33]
dawagner: we currently disallow compound CSS selectors, not disallowed by the spec.
wilhelm: clarifies that specifying a css selector "h1,h2" will return elements of either type.
<scribe> ACTION: simonstewart : allow compound css selectors. Get rid of class selectors. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action34]
<scribe> ACTION: section 9.2.5, spell out availability of functionality against html and xhtml [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action35]
<scribe> ACTION: section 9.2.5 drop step 3 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action36]
simonstewart: 9.2.7 aria - using the accessibility API.
<scribe> ACTION: write section 9.2.7 , if you're familiar with the spec. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action37]
wilhelm: we have 21.5 minutes left
<scribe> ACTION: dawagner : visibility ancestry is different than dom ancestry [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action38]
jleyba: section 10 bullet 3, is in accurate.
<scribe> ACTION: dawagner : follow up on 10, last bullet point - element inside a fixed-sized parent, outside the size. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action39]
discussion about properties, attributes, regarding section 10.1
jari, dawagner : we should not conflate attributes and properties.
<scribe> ACTION: 10.1, re-work this section so that only style and aliases are left over. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action40]
<jari> slightly relevant section of the HTML spec on properties vs attributes: http://www.whatwg.org/specs/web-apps/current-work/#reflecting-content-attributes-in-idl-attributes
dawagner, we could do aliasing in the client libraries. simonstewart : let's leave in the spec
<scribe> ACTION: section 10.1.2.1 css should always be lower case, colour should always be rgba. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action41]
<jari> scribe: jari
<scribe> ACTION: 10.1.1 - reword to be consistent with the HTML spec (see http://www.whatwg.org/specs/web-apps/current-work/#reflecting-content-attributes-in-idl-attributes) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action42]
jleyba: what about non-standard boolean attributes?
dawagner: example: "disabled" on input elements different from "disabled" on a div
<scribe> ACTION: 10.1.1 - general cleanup [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action43]
simonstewart: (reading section
... HTML4 and HTML5 differs in the boolean value of disabled="cheese" and disabled="disabled"
<scribe> ACTION: wilhelm investigates HTML4 vs HTML5 and boolean attributes (esp. disabled on input vs non-input elements) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action44]
simonstewart: moving to section 10.1.3
<scribe> ACTION: getElementAttribute algorithm should be explicit why we're special casing URLs (reason: always return absolute URLs) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action45]
<scribe> ACTION: 10.1.3 - "algorithm defined above" should be named and linked [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action46]
<wilhelm> + iframes, frame, link, map, area
simonstewart: moving on to 10.2
<davidburns> I am heading out, will try join in about 15 mins again
reading through the algorithm for getting the visible text of an element
<dawagner> ACTION: dawagner: Fix 1.1 of the rendering text algorithm [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action47]
jimevans: is there need for a command to determine the presence of an element?
jleyba: the efficient way to do this has already been established
not going to be added.
<scribe> ACTION: use "DOMString" WebIDL type over "string" [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action48]
jleyba: (reading section 11.1, 11.2, 11.3)
dawagner: 11.1: Argument type; dictionary should be of Argument types
<scribe> ACTION: limit recursion depth in recursive algorithms (a SHOULD) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action49]
simonstewart: should return null to user and browser is in undefined state
simonstewart: should be able to use captured logs
jleyba: if there's an unhandled error and you haven't queried for it, the next command should fail
jleyba: could be optionally
... similar to how alerts are handled
... webdriver perfectly capable of propagating failure when JS errors occur
simonstewart: not sure; gut
feeling, seems like overkill
... not enough people care about this
leaving this as an open question
dawagner: could have more specific type than UnknownError when we know what's happening (Section 11.3 step 2)
jleyba: could also not throw and just return null
dawagner: makes more sense to throw
<scribe> ACTION: link to DOM4 instead of DOM3 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action50]
This is scribe.perl Revision: 1.136 of Date: 2011/05/12 12:01:43 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/always/already/ Found Scribe: simonstewart Inferring ScribeNick: simonstewart Found Scribe: wilhelm Inferring ScribeNick: wilhelm Found Scribe: jari Inferring ScribeNick: jari Found Scribe: eranm Inferring ScribeNick: eranm Found Scribe: jari Inferring ScribeNick: jari Scribes: simonstewart, wilhelm, jari, eranm ScribeNicks: simonstewart, wilhelm, jari, eranm WARNING: No "Present: ... " found! Possibly Present: MikeSmith_ SamitBadle abarsto andreastt chwells danielwh darobin davidburns dawagner eranm freynaud freynaud_ guest guest2 hugs jari jarib jimevans jleyba kevin klm kmenard others plh plh_ pmfji raga rambighananthan samit simonstewart timeless wilhelm You can indicate people for the Present list like this: <dbooth> Present: dbooth jonathan mary <dbooth> Present+ amy Got date from IRC log name: 19 Apr 2012 Guessing minutes URL: http://www.w3.org/2012/04/19-testing-minutes.html WARNING: No person found for ACTION item: in section 6.4, take out the equivalence to quit. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action15] People with action items: - 10.1 10.1.1 10.1.3 2.1 5 7.3 9.2.2. 9.2.4 add andreastt are be change changed clarify clauses commands dawagner define document eranm fix getelementattribute go jleyba limit link maximise navigating re-phrase reword safe say section server should simonstewart specify step swap switch throws use we webelementlist which wilhelm write WARNING: Input appears to use implicit continuation lines. You may need the "-implicitContinuations" option.[End of scribe.perl diagnostic output]