This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.
For the Navigation Timing tests, it is not accurate to group Safari and Chrome into one WebKit engine. The summary page makes it look like WebKit often fails the tests: http://w3c-test.org/framework/results/nav-timing-default/ However, drilling into one test it becomes clear what is going on: http://w3c-test.org/framework/details/nav-timing-default/test_navigate_within_document/engine/webkit/ All Chrome runs pass and all Safari runs fail. So the test result is really just indicating whether the test was run more often in Safari or Chrome. This needs to be fixed to separate them out. I suggest columns "WebKit (Safari)" and "WebKit (Chrome)". WebKit variants other than Safari/Chrome should be sufficiently rare to discard. I suspect this may be true for test suites other than Navigation Timing, but haven't verified. Very often there is browser code outside of WebKit that is necessary to support web platform APIs. It is also very common to have flag guards to enable/disable web platform APIs in WebKit.
Is this a report formatting issue or for these tests do you really consider Chrome and Safari to be independent implementations? In other words, if your CR exit criteria are two independent implementations, and the only passes you have are WebKit (Safari) and WebKit (Chrome), does that meet your exit criteria? FWIW, this is not the case in the vast majority of the cases. Also, the intent of the current report is to show the status of meeting CR exit criteria. As all results are tracked by UA string, and engines are tracked independently of browsers, it would be easy to make a different report that shows results per browser and leave the current report alone.
(In reply to comment #1) > Is this a report formatting issue or for these tests do you really consider > Chrome and Safari to be independent implementations? For navigation timing, half of the implementation is in WebKit. The other half is in the browser's network stack which is implemented separately by each port of WebKit. Currently Chrome has it implemented and thus the API is flipped on in WebKit. Safari has not implemented that side so the API is disabled in WebKit. > In other words, if your CR exit criteria are two independent implementations, > and the only passes you have are WebKit (Safari) and WebKit (Chrome), does that > meet your exit criteria? No. You are right, I didn't think of it that way. However, the converse is equally concerning. If we had run the test more often in Safari than Chrome, it would appear that we didn't have a conforming engine at all. > Also, the intent of the current report is to show the status of meeting CR exit > criteria. As all results are tracked by UA string, and engines are tracked > independently of browsers, it would be easy to make a different report that > shows results per browser and leave the current report alone. I'd be in favor of a second report that separates out browsers. If for no other reason than to feed bloggers and the like who enjoy quoting numbers like "Browser X passes Y% of conformance tests." However, that doesn't necessarily fully address the concern above.
(In reply to comment #2) > (In reply to comment #1) > > In other words, if your CR exit criteria are two independent implementations, > > and the only passes you have are WebKit (Safari) and WebKit (Chrome), does that > > meet your exit criteria? > > No. You are right, I didn't think of it that way. However, the converse is > equally concerning. If we had run the test more often in Safari than Chrome, it > would appear that we didn't have a conforming engine at all. The relative number of results are not relevant. Currently, if there's one pass for an engine, any version, any browser, that engine is considered passed, regardless of how many other results there may be (so long as none of the results are indications that the test itself is invalid). Results where there are passes, but also other results are reported in a lighter shade of green. At some point we may add a better filtering algorithm, but what we have now has worked well for the CSS WG. Either way, it'll never be "more passes than fails", it would compare versions of engines, browsers, etc. and try to eliminate false pass results. > > > Also, the intent of the current report is to show the status of meeting CR exit > > criteria. As all results are tracked by UA string, and engines are tracked > > independently of browsers, it would be easy to make a different report that > > shows results per browser and leave the current report alone. > > I'd be in favor of a second report that separates out browsers. If for no other > reason than to feed bloggers and the like who enjoy quoting numbers like > "Browser X passes Y% of conformance tests." > > However, that doesn't necessarily fully address the concern above. I suggest closing this issue, since this report is really only about CR exit criteria, and filing a feature request for a per-browser result report. There are also filters available internally (you can currently get them by hand-editing the report URI) to generate reports for specific browsers, platforms and versions of engine, browser and platform, but no UI yet to select them, that UI needs to be added as well.