27097 2014-10-18 13:37:01 +0000 Mandate ordering which required capabilities must be checked 2014-10-22 23:51:36 +0000 1 1 1 Unclassified Browser Test/Tools WG WebDriver unspecified PC All NEW P2 normal --- 1 ato public-browser-tools-testing dburns mike vlotoshnikov public-browser-tools-testing oldest_to_newest 113414 0 ato 2014-10-18 13:37:01 +0000 In section 4.1.1 we should mandate, and not merely suggest, the ordering in which the required capabilities must be checked. This is important for interoperability so that one remote end doesn't parse browserVersion before browserName. The mandated order should be: 1. browserName 2. browserVersion 3. platformName 4. platformVersion 113488 1 dburns 2014-10-20 20:11:32 +0000 Since we are should check them all, should we even have this note? I copied this across but thinking about it I think we should be returning all capabilites that do not match in the error message(this is currently missing from the prose and I need to add it). Since all errors are being returned what value would there be (for interop) in mandating the order of capabilities? 113491 2 dburns 2014-10-20 20:58:05 +0000 https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=e1e1b23f7739 is an example of what I mean, patch isnt complete but at least it's working 113558 3 vlotoshnikov 2014-10-22 10:59:46 +0000 In certain cases, when a remote returns an error, it might not be very clear which capabilities are or aren't met. Example: an intermediary remote node (selenium-java-server/grid) receives a startSession with required browserName=firefox and browserVersion=20. But it does not have any firefoxes available. So it knows that browserName is a cap that it cannot match. browserVersion=20: was its presense or value a reason for it to return an error response? No, it would return an error regardless. Should it include browserVersion as a capability it couldn't match? Unclear. 113559 4 vlotoshnikov 2014-10-22 11:17:02 +0000 Re the patch (full file: https://hg.mozilla.org/try/file/dc7d0afee49c/testing/marionette/marionette-server.js ). That code will handle unknown (to that driver) capabilities in a way that looks both non-compliant and sub-optimal: it will include all unknown capabilities in this.sessionCapabilities and (I assume) will be returning them as "actual capabilities" of the session. So the local end may check the returned actual capabilities and conclude the capability it asked for (required) is met - while marionette doesn't know anything about it and likely doesn't actually provide it. I agree with Andreas who suggested that drivers MUST ignore requested capabilities they don't know about***. And I think drivers definitely must not return capabilities they do not know about (and they should assume they therefore don't provide them) as "actual capabilities" of the session. *** "know about" is a tricky concept, has edge cases, the spec should define that very specifically. 113562 5 vlotoshnikov 2014-10-22 11:41:22 +0000 When we include intermediary remote end nodes in the picture of matching capabilities and returning actual capabilities for a session, it becomes a little tricky, with some hard to reconcile edge cases. (I could give a couple of examples) Not sure how deeply into intermediary-nodes considerations the spec should go. 113622 6 dburns 2014-10-22 23:51:36 +0000 (In reply to seva from comment #3) > In certain cases, when a remote returns an error, it might not be very clear > which capabilities are or aren't met. Example: an intermediary remote node > (selenium-java-server/grid) receives a startSession with required > browserName=firefox and browserVersion=20. But it does not have any > firefoxes available. So it knows that browserName is a cap that it cannot > match. browserVersion=20: was its presense or value a reason for it to > return an error response? No, it would return an error regardless. Should it > include browserVersion as a capability it couldn't match? Unclear. I agree that all errors should be returned, so ALL capabilities passed in should be tested and if they all fail then all of them should be in the error message returned to the local end. (In reply to seva from comment #4) > I agree with Andreas who suggested that drivers MUST ignore requested > capabilities they don't know about***. And I think drivers definitely must > not return capabilities they do not know about (and they should assume they > therefore don't provide them) as "actual capabilities" of the session. The reason why I left it in is that I think that I can see a use case where the local end might want to store information in the capabilities. E.g. test run number, revision that the session was created against and then use the capabilites for a bit of state. (In reply to seva from comment #5) > When we include intermediary remote end nodes in the picture of matching > capabilities and returning actual capabilities for a session, it becomes a > little tricky, with some hard to reconcile edge cases. (I could give a > couple of examples) > > Not sure how deeply into intermediary-nodes considerations the spec should > go. We should probably specify this :)