Re: web-platform-tests results

On Wed, Dec 19, 2018 at 9:27 AM Philippe Le Hégaret <plh@w3.org> wrote:

> On 12/17/2018 8:23 AM, Philippe Le Hégaret wrote:
> >> Also you should be able to test with safari now with their preview
> >> announcement the other day.
> >
> > I don't have the proper computer for that unfortunately. If someone else
> > has the preview, a key, and a python install, I'm happy to do remote
> > assistance to get those results.
>
> Sam and I managed to generate test results for safari tech preview 71.
>
> Added in
>   https://w3c.github.io/test-results/webauthn/less-than-2.html
>   https://w3c.github.io/test-results/webauthn/all.html
>
> This is outside the scope of the spec but note that Safari doesn't
> prompt the user to use a key and doesn't active the key if it has a pin.
> We had to use a key that did not have a pin set.
>

I've reviewed every test that Chrome was listed as failing. Overall, *I
don't believe that the spec should be held up by these test results*
because I don't believe that the spec need be changed because of them.

There are a number of tests which are failing basically everywhere:

Bad AuthenticatorSelectionCriteria: authenticatorSelection is empty array
> Bad AuthenticatorSelectionCriteria: authenticatorSelection is null
> Bad AuthenticatorSelectionCriteria: authenticatorSelection residentKey is
> string
> Bad rp: id is object
> Bad rp: name is null
> Bad rp: name is object
> Bad user: displayName is null
> Bad user: displayName is object
> Bad user: name is null
> Bad user: name is object
> Bad extensions: extensions is null
> Bad extensions: extensions is empty Array
> Bad extensions: extensions is empty ArrayBuffer


These are putting values of the wrong type into various fields. However,
the ECMAScript to WebIDL conversions are very forgiving and I believe these
tests are invalid. For example, consider what values are valid for a WebIDL
boolean <https://heycam.github.io/webidl/#es-boolean>. This references TC39
7.1.2 <https://tc39.github.io/ecma262/#sec-toboolean> which considers,
amongst other things, both {} and [] to be "true".

Bad AuthenticatorSelectionCriteria: authenticatorSelection residentKey true

Chromium does not support resident keys until privacy issues are addressed,
but this is a CTAP2 issue.

Bad rp: icon is null
> Bad rp: icon is empty String
> Bad rp: icon is object
> Bad user: icon is empty String
> Bad user: icon is null
> Bad user: icon is object

These are arguably semantic errors due to the requirements of |icon|.
However, Chrome does not currently implement icon.

Bad rp: name is empty String

Bad user: name is empty String

 I don't see a prohibition on it being empty.

Bad user: ArrayBuffer id is too long (65 bytes)

(and similar)

Bug in Chromium. Will be fixed by
https://chromium-review.googlesource.com/c/chromium/src/+/1385104

Bad user: id is empty ArrayBuffer

We agreed on the call of 2018-12-19 that this test should be deleted.

exclude existing credential

I think the test is wrong. It's expecting "NotAllowedError" but the spec
says it should be InvalidStateError by my reading.

Bad extensions: extensions is null
> Bad extensions: extensions is empty Array
> Bad extensions: extensions is empty ArrayBuffer
> Bad extensions: malformatted JSON
> Bad extensions: JavaScript object
> Bad extensions: extension ID too long

Assumes that browsers are parsing all extensions and doing the generic
transform to CBOR. I don't believe that any browser intends to do this.
Arguably the over-long extension ID could be caught though.

passing credentials.create() with rpId (host and port)
> passing credentials.get() with rpId (host and port)

I believe this test is incorrect because RP IDs do not include a port
number. (See “It does not itself include a scheme or port” in the spec.)

DataView user id

Works in current Chrome build

Bad pubKeyCredParams: pubKeyCredParams is empty Array

Current Chrome fails this although it was marked as “passing”. It does
appear that it's valid for this sequence to be empty. Thus I believe this
is a Chrome bug.

Bad pubKeyCredParams: first param has bad alg (42)
> Bad pubKeyCredParams: first param has bad alg (0)

I think this test is wrong. It's not invalid to specify an unknown
COSEAlgorithmIdentifier: maybe the user will insert a token that
understands it.

no credential specified

This failure ("Attempting list without defining credential to test") is
coming from within the test itself (helper.js).


Cheers

AGL