Web Cryptography Working Group Teleconference

25 Mar 2016

See also: IRC log




<scribe> scribe: hhalpin

Mike: You can safely ignore the CSS work, its in a different repo
... the metadata on the test
... we just have javascript-based testharness.js
... that let you write assertions
... you use assertions (pass/fails)
... with some kind of indication of what failed
... you can spend a bunch of time looking at docs and getting an idea
... by writing a test-case
... the more effective way is w3 IRC #testing

<MikeSmith> jgraham

<MikeSmith> Ms2ger

Mike: when I have a question I tend to go to IRC and ask those two

So input is the WebIDL, evaluates all contraints

long story short: YES

the test system is designed so you can run it locally

you don't have to run it on remote web-server

We want people who are writing this from scratch to be the ideal audience

things like regressions

is our real goal so that its integrated into every single browser CI

so they run all these tests every time they commit

so there are no unexpected results

Mozilla does this

all browser vendors to be publishing their results

typical exit criteria every single feature has at least two independent implementations in browser issues

the bad thing is that its solving the problems for implementers for the test-suite to be extremely thorough

that covers every requirement

In short-term get WebIDL testing started first

then go through a higher-level

We need to go through the whole reviewer

for each PR, the owner the owner gets notified

our expectation is that you self-manage your own directory

jimsch: We need the WebIDL test
... then we do some sort of coverage test
... then go to corner cases
... I'm not sure how to get to this 'server' and set-it up on my system

MikeSmith: If you've ever used Python has a built in webserver
... browser into directory on shell
... type in pythonsimple http (sp?)
... then you get a browser with a localhost
... so our test-suite is layered on that
... once you check out that repo
... change directory in shell to that directory
... ./serve.py
... that serves up python webserver
... and exposes the entire test-suite
... on your local webserver
... we added some hooks to allow you to easily add headers

a '.headers' file in same directory

scribe: and you can then use python
... so it runs when you explicitly call it, to stop it you gotta CTRL-C
... the one thing to note is that you have to edit the /etc/hosts
... or its equivalent
... web-platform.tests
... (sp?)
... adds other domain names
... for cross-origin testing
... then you just shut things down

<MikeSmith> https://en.wikipedia.org/wiki/Hosts_(file)#Location_in_the_file_system

<MikeSmith> %SystemRoot%\System32\drivers\etc\hosts

<apowers> I would love to hear about best practices too, for the Web Authentication tests

<MikeSmith> https://github.com/w3c/web-platform-tests/blob/master/README.md#running-the-tests

<wseltzer> MikeSmith: Best practice to start with the W3C TTWF platform

<wseltzer> apowers: I'm working on WebAuthn testing

<wseltzer> MikeSmith: are you using testharness.js?

<wseltzer> apowers: yes, I've written about 4 tests of attribute presence so far

<wseltzer> ... similar questions about level of detail

<wseltzer> MikeSmith: Some people have ported existing tests

<wseltzer> ... a few folks on each of the browser projects

<wseltzer> ... also, in each browser, to get code to trunk, you need tests

<wseltzer> ... so at a minimum, you need that in crypto test suite

<wseltzer> ... our policy, if you have tests that have been reviewed and approved in implementation

<wseltzer> ... for a feature, then those can go into the tree of WPT without any additional reveiw needed

<wseltzer> ... so we should look at getting existing tests into the repo

<wseltzer> ... if in a browser project, tests are sufficient for your needs, that's probably useful for us

<apowers> just to be sure I'm headed in the right direction, this was my first test to make sure my tests would run as part of the web-platform-tests: https://github.com/apowers313/w3c-auth-test

<wseltzer> ... then fill in the holes

<wseltzer> ... best practices on coverage: the best you can have

<wseltzer> apowers: different styles, etc.

<wseltzer> MikeSmith: yes, not everything in there is a good example

<wseltzer> ... so if you're wondering whether a given test is good, ask us on irc #testing

<wseltzer> ... or by looking at the commit log

<MikeSmith> https://github.com/w3c/web-platform-tests/graphs/contributors

<wseltzer> ... if you see jgraham, ms2ger, zcorpan (Simon Pieters, Opera)

<wseltzer> ... sideshowbarker (Mike Smith)

<wseltzer> hhalpin: how do we test features that, e.g., require a certain format key, when we don't have the input

<wseltzer> ... take it out because we can't test it?

<wseltzer> ... make sure it's non-normative?

<wseltzer> ... or assume it works?

<wseltzer> MikeSmith: testharness.js can't test anything that's not programmatically exposed to JS

<wseltzer> ... you can't automate with testharness.js if it's not exposed at the JS layer

<wseltzer> ... you may be able to do it with python hooks, e.g. for HTTP headers

<wseltzer> ... in that case, write a manual test

<wseltzer> ... with instructions to the user how to run it

<wseltzer> ... but the problem is that no one ever runs manual tests; not integrated into CI

<wseltzer> ... don't remove requirements from the spec just because you can't test them automatically in JS

<wseltzer> ... we do have a webdriver spec, selenium webdriver

<wseltzer> ... if you can't even write a manual test, that's a concern

<wseltzer> jimsch: I'm not worried aobut the key issues, I've started writing tests there

<wseltzer> jimsch++

<wseltzer> jimsch: naming of tests. any standard method of giving tests unique names?

<wseltzer> MikeSmith: in CSS, we used a simple hierarchical naming strategy for filenames

<wseltzer> ... no requirement of global uniqueness

<wseltzer> ... just has to be unique in its directory

<jimsch> Is that the name of the file or the name passed into async_test?

<wseltzer> ... so all the top-level idreactories are spec names

<wseltzer> ... individual directories for spec sections

<wseltzer> ... filenames matched roughly to the requirements

<wseltzer> jimsch: workers. Anything special to consider there?

essentially,Mike's message was that the names do not need to be unique

re files in test-suite

<MikeSmith> Ms2ger

did the Workers work.

<wseltzer> hhalpin: 2 big issues were keys and workers

<wseltzer> MikeSmith: Web workers is stable, shipped

to be clear *Service Workers*

<wseltzer> ... Service Workers are different. newer, much less stable

<wseltzer> ... currently only in FF and chrome

<wseltzer> RyanHurst: Concern was implementation status

I believe the answer is FF did commit (although not yet done) to enable WebCrypto on ServiceWorkers

earlier that was unclear

jimsch: essentia lly, sleevi wants the same tests for normal tests also run as service workers

<wseltzer> jimsch: is there support for SW in the runners?

<jimsch> is this the right way to do things? https://github.com/w3c/web-platform-tests/blob/master/WebCryptoAPI/getRandomValues.worker.js

<wseltzer> MikeSmith: I don't think we have convenience methods, but the code should run for SWs

i.e. can we just hit a magic importScripts()

<wseltzer> MikeSmith: looking at getRandomValues, that's not how I'd do it

<apowers> is there any documentation for the process of how new tests end up in the web-platform-tests repo? e.g. - setting up a new repo, submitting PRs, approval processes, where work should occur, etc.?

I think https://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#importscripts

<wseltzer> MikeSmith: most tests are written using Script tgs

he seems to think that invokes a service worker

<wseltzer> MikeSmith: oh, there's a separate .html

<wseltzer> hhalpin: welcome Ryan

<wseltzer> Ryan_Hurst: recently joined Google

there was all this Discourse stuff for example

i.e. test-suite review

<MikeSmith> https://github.com/w3c/web-platform-tests/blob/master/WebCryptoAPI/OWNERS

<wseltzer> MikeSmith: We can add to the OWNERS file

<wseltzer> ... owners get notified when someone sends a PR on the test directory

So we want to add Jim and Ryan to the owners file - can you throw me your github ids?

<jimsch> jimsch

<wseltzer> MikeSmith: You're not required ot use critic code review

<wseltzer> ... no hard requirements on how you do review; you do need to review code

I think using Github by itself is just fine

<wseltzer> ... and document that you've reviewed

<jimsch> +1 to using github for now

<Engelke> I have to go now. I'm going to try to create a test by next Wednesday.

<jimsch> What test are you going to try for?

Thanks Charles!

<wseltzer> jimsch: as reviewer, should I clone and run the tests?

<wseltzer> MikeSmith: Yes

<wseltzer> ... clone the PR branch and run

<wseltzer> git-checkout #

<MikeSmith> http://w3c-test.org/submissions/

<wseltzer> MikeSmith: Every PR has the state of the PR branch ^

<wseltzer> ... I just tend to check out locally

<wseltzer> hhalpin: If we get WebIDL, keys, and then figure out service workers with ms2ger

<wseltzer> MikeSmith: I recommend that the group and owners set a realistic timeline

<wseltzer> ... that will help me to help you

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.144 (CVS log)
$Date: 2016/03/25 17:13:34 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/If you/MikeSmith: If you/
Succeeded: s/@@/RyanHurst/
Found Scribe: hhalpin
Inferring ScribeNick: hhalpin

WARNING: No "Topic:" lines found.

WARNING: No "Present: ... " found!
Possibly Present: Engelke Mike MikeSmith RyanHurst Ryan_Hurst apowers hhalpin https jimsch tantek trackbot wseltzer
You can indicate people for the Present list like this:
        <dbooth> Present: dbooth jonathan mary
        <dbooth> Present+ amy

WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth

Found Date: 25 Mar 2016
Guessing minutes URL: http://www.w3.org/2016/03/25-crypto-minutes.html
People with action items: 

WARNING: No "Topic: ..." lines found!  
Resulting HTML may have an empty (invalid) <ol>...</ol>.

Explanation: "Topic: ..." lines are used to indicate the start of 
new discussion topics or agenda items, such as:
<dbooth> Topic: Review of Amy's report

[End of scribe.perl diagnostic output]