15:17:37 <RRSAgent> RRSAgent has joined #crypto
15:17:37 <RRSAgent> logging to http://www.w3.org/2016/03/25-crypto-irc
15:17:39 <trackbot> RRSAgent, make logs public
15:17:41 <trackbot> Zakim, this will be CRYPT
15:17:41 <Zakim> I do not see a conference matching that name scheduled within the next hour, trackbot
15:17:42 <trackbot> Meeting: Web Cryptography Working Group Teleconference
15:17:42 <trackbot> Date: 25 March 2016
15:18:30 <tantek> tantek has joined #crypto
15:19:44 <hhalpin> scribe: hhalpin
15:20:01 <hhalpin> Mike: You can safely ignore the CSS work, its in a different repo
15:20:09 <hhalpin> ... the metadata on the test
15:20:27 <hhalpin> ... we just have javascript-based testharness.js
15:20:33 <hhalpin> ... that let you write assertions
15:20:54 <hhalpin> ... you use assertions (pass/fails)
15:21:08 <hhalpin> ... with some kind of indication of what failed
15:21:39 <hhalpin> ... you can spend a bunch of time looking at docs and getting an idea
15:22:00 <hhalpin> ... by writing a test-case
15:22:15 <hhalpin> ... the more effective way is w3 IRC #testing
15:22:42 <MikeSmith> jgraham
15:22:48 <MikeSmith> Ms2ger
15:24:04 <hhalpin> ... when I have a question I tend to go to IRC and ask those two
15:24:15 <MikeSmith> q?
15:24:18 <hhalpin> q+
15:26:03 <hhalpin> So input is the WebIDL, evaluates all contraints
15:28:47 <hhalpin> long story short: YES
15:28:57 <hhalpin> the test system is designed so you can run it locally
15:29:07 <hhalpin> you don't have to run it on remote web-server
15:31:37 <hhalpin> We want people who are writing this from scratch to be the ideal audience
15:31:43 <hhalpin> things like regressions
15:31:56 <hhalpin> is our real goal so that its integrated into every single browser CI
15:32:01 <hhalpin> so they run all these tests every time they commit
15:32:06 <hhalpin> so there are no unexpected results
15:32:08 <hhalpin> Mozilla does this
15:32:27 <hhalpin> all browser vendors to be publishing their results
15:33:22 <hhalpin> typical exit criteria every single feature has at least two independent implementations in browser issues
15:33:54 <hhalpin> the bad thing is that its solving the problems for implementers for the test-suite to be extremely thorough
15:33:57 <hhalpin> that covers every requirement
15:34:52 <hhalpin> In short-term get WebIDL testing started first
15:34:58 <hhalpin> then go through a higher-level
15:35:39 <hhalpin> RRSAgent, draft minutes
15:35:39 <RRSAgent> I have made the request to generate http://www.w3.org/2016/03/25-crypto-minutes.html hhalpin
15:35:58 <hhalpin> We need to go through the whole reviewer
15:36:12 <hhalpin> for each PR, the owner the owner gets notified
15:36:33 <hhalpin> our expectation is that you self-manage your own directory
15:37:12 <MikeSmith> q?
15:37:18 <hhalpin> ack hhalpin
15:37:26 <hhalpin> jimsch: We need the WebIDL test
15:37:31 <hhalpin> ... then we do some sort of coverage test
15:37:35 <hhalpin> ... then go to corner cases
15:38:05 <hhalpin> ... I'm not sure how to get to this 'server' and set-it up on my system
15:38:25 <hhalpin> If you've ever used Python has a built in webserver
15:38:33 <hhalpin> ... browser into directory on shell
15:38:46 <hhalpin> ... type in pythonsimple http (sp?)
15:38:48 <wseltzer> s/If you/MikeSmith: If you/
15:38:56 <hhalpin> ... then you get a browser with a localhost
15:39:28 <hhalpin> ... so our test-suite is layered on that
15:39:33 <hhalpin> ... once you check out that repo
15:39:43 <hhalpin> ... change directory in shell to that directory
15:39:49 <hhalpin> ... ./serve.py
15:39:55 <hhalpin> ... that serves up python webserver
15:40:02 <hhalpin> ... and exposes the entire test-suite
15:40:07 <hhalpin> ... on your local webserver
15:40:16 <hhalpin> ... we added some hooks to allow you to easily add headers
15:40:27 <hhalpin> a '.headers' file in same directory
15:40:58 <hhalpin> ... and you can then use python
15:41:08 <hhalpin> ... so it runs when you explicitly call it, to stop it  you gotta CTRL-C
15:41:26 <hhalpin> ... the one thing to note is that you have to edit the /etc/hosts
15:41:31 <hhalpin> ... or its equivalent
15:41:41 <hhalpin> ... web-platform.tests
15:41:45 <hhalpin> ... (sp?)
15:41:49 <hhalpin> ... adds other domain names
15:42:02 <hhalpin> ... for cross-origin testing
15:42:16 <hhalpin> ... then you just shut things down
15:44:40 <MikeSmith> https://en.wikipedia.org/wiki/Hosts_(file)#Location_in_the_file_system
15:45:07 <MikeSmith> %SystemRoot%\System32\drivers\etc\hosts
15:45:56 <apowers> I would love to hear about best practices too, for the Web Authentication tests
15:46:32 <MikeSmith> https://github.com/w3c/web-platform-tests/blob/master/README.md#running-the-tests
15:47:13 <wseltzer> MikeSmith: Best practice to start with the W3C TTWF platform
15:47:58 <wseltzer> apowers: I'm working on WebAuthn testing
15:48:05 <wseltzer> MikeSmith: are you using testharness.js?
15:48:27 <wseltzer> apowers: yes, I've written about 4 tests of attribute presence so far
15:48:50 <wseltzer> ... similar questions about level of detail
15:49:12 <wseltzer> MikeSmith: Some people have ported existing tests
15:49:35 <wseltzer> ... a few folks on each of the browser projects
15:49:56 <wseltzer> ... also, in each browser, to get code to trunk, you need tests
15:50:15 <wseltzer> ... so at a minimum, you need that in crypto test suite
15:50:31 <wseltzer> ... our policy, if you have tests that have been reviewed and approved in implementation
15:50:47 <wseltzer> ... for a feature, then those can go into the tree of WPT without any additional reveiw needed
15:51:26 <wseltzer> ... so we should look at getting existing tests into the repo
15:52:25 <wseltzer> ... if in a browser project, tests are sufficient for your needs, that's probably useful for us
15:52:28 <apowers> just to be sure I'm headed in the right direction, this was my first test to make sure my tests would run as part of the web-platform-tests: https://github.com/apowers313/w3c-auth-test
15:52:39 <wseltzer> ... then fill in the holes
15:53:02 <wseltzer> ... best practices on coverage: the best you can have
15:53:46 <wseltzer> apowers: different styles, etc.
15:54:06 <wseltzer> MikeSmith: yes, not everything in there is a good example
15:54:32 <wseltzer> ... so if you're wondering whether a given test is good, ask us on irc #testing
15:54:38 <wseltzer> ... or by looking at the commit log
15:55:09 <MikeSmith> https://github.com/w3c/web-platform-tests/graphs/contributors
15:55:34 <wseltzer> ... if you see jgraham, ms2ger, zcorpan (Simon Pieters, Opera)
15:55:42 <wseltzer> ... sideshowbarker (Mike Smith)
15:58:02 <wseltzer> hhalpin: how do we test features that, e.g., require a certain format key, when we don't have the input
15:58:27 <wseltzer> ... take it out because we can't test it?
15:58:34 <wseltzer> ... make sure it's non-normative?
15:58:40 <wseltzer> ... or assume it works?
15:59:03 <wseltzer> MikeSmith: testharness.js can't test anything that's not programmatically exposed to JS
15:59:20 <wseltzer> ... you can't automate with testharness.js if it's not exposed at the JS layer
15:59:38 <wseltzer> ... you may be able to do it with python hooks, e.g. for HTTP headers
15:59:53 <wseltzer> ... in that case, write a manual test
16:00:02 <wseltzer> ... with instructions to the user how to run it
16:00:36 <wseltzer> ... but the problem is that no one ever runs manual tests; not integrated into CI
16:00:51 <wseltzer> ... don't remove requirements from the spec just because you can't test them automatically in JS
16:01:02 <wseltzer> ... we do have a webdriver spec, selenium webdriver
16:01:47 <wseltzer> ... if you can't even write a manual test, that's a concern
16:02:02 <wseltzer> jimsch: I'm not worried aobut the key issues, I've started writing tests there
16:02:08 <wseltzer> jimsch++
16:02:33 <wseltzer> jimsch: naming of tests. any standard method of giving tests unique names?
16:03:10 <wseltzer> MikeSmith: in CSS, we used a simple hierarchical naming strategy for filenames
16:03:17 <wseltzer> ... no requirement of global uniqueness
16:03:29 <wseltzer> ... just has to be unique in its directory
16:03:34 <jimsch> Is that the name of the file or the name passed into async_test?
16:03:40 <wseltzer> ... so all the top-level idreactories are spec names
16:03:49 <wseltzer> ... individual directories for spec sections
16:03:59 <wseltzer> q+ jimsch
16:04:36 <wseltzer> ... filenames matched roughly to the requirements
16:05:18 <wseltzer> ack jimsch
16:07:19 <wseltzer> jimsch: workers. Anything special to consider there?
16:07:27 <hhalpin> essentially,Mike's message was that the names do not need to be unique
16:07:32 <hhalpin> re files in test-suite
16:07:35 <MikeSmith> Ms2ger
16:07:40 <hhalpin> did the Workers work.
16:08:42 <wseltzer> hhalpin: 2 big issues were keys and workers
16:09:16 <wseltzer> MikeSmith: Web workers is stable, shipped
16:09:24 <hhalpin> to be clear *Service Workers*
16:09:57 <wseltzer> ... Service Workers are different. newer, much less stable
16:10:19 <wseltzer> ... currently only in FF and chrome
16:10:52 <wseltzer> @@: Concern was implementation status
16:10:53 <hhalpin> I believe the answer is FF did commit (although not yet done) to enable WebCrypto on ServiceWorkers
16:11:17 <hhalpin> earlier  that was unclear
16:11:41 <hhalpin> jimsch: essentia lly, sleevi wants the same tests for normal tests also run as service workers
16:14:11 <wseltzer> jimsch: is there support for SW in the runners?
16:14:24 <jimsch> is this the right way to do things? https://github.com/w3c/web-platform-tests/blob/master/WebCryptoAPI/getRandomValues.worker.js
16:15:03 <wseltzer> MikeSmith: I don't think we have convenience methods, but the code should run for SWs
16:16:12 <hhalpin> i.e. can we just hit a magic importScripts()
16:17:50 <wseltzer> MikeSmith: looking at getRandomValues, that's not how I'd do it
16:18:14 <apowers> is there any documentation for the process of how new tests end up in the web-platform-tests repo? e.g. - setting up a new repo, submitting PRs, approval processes, where work should occur, etc.?
16:18:25 <hhalpin> I think https://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#importscripts
16:18:30 <wseltzer> MikeSmith: most tests are written using Script tgs
16:19:11 <hhalpin> he seems to think that invokes a service worker
16:20:23 <wseltzer> MikeSmith: oh, there's a separate .html
16:21:31 <wseltzer> hhalpin: welcome Ryan
16:21:57 <wseltzer> Ryan_Hurst: recently joined Google
16:22:11 <hhalpin> there was all this Discourse stuff for example
16:22:16 <hhalpin> i.e. test-suite review
16:22:34 <MikeSmith> https://github.com/w3c/web-platform-tests/blob/master/WebCryptoAPI/OWNERS
16:22:43 <wseltzer> MikeSmith: We can add to the OWNERS file
16:23:06 <wseltzer> ... owners get notified when someone sends a PR on the test directory
16:23:18 <hhalpin> So we want to add Jim and Ryan to the owners file - can you throw me your github ids?
16:24:08 <jimsch> jimsch
16:24:53 <wseltzer> MikeSmith: You're not required ot use critic code review
16:25:52 <wseltzer> ... no hard requirements on how you do review; you do need to review code
16:25:54 <hhalpin> I think using Github by itself is just fine
16:26:48 <hhalpin> s/@@/RyanHurst
16:26:57 <wseltzer> ... and document that you've reviewed
16:27:20 <jimsch> +1 to using github for now
16:27:50 <Engelke> I have to go now. I'm going to try to create a test by next Wednesday.
16:28:07 <jimsch> What test are you going to try for?
16:28:09 <hhalpin> Thanks Charles!
16:29:41 <wseltzer> jimsch: as reviewer, should I clone and run the tests?
16:29:43 <wseltzer> MikeSmith: Yes
16:29:55 <wseltzer> ... clone the PR branch and run
16:30:32 <wseltzer> git-checkout #
16:30:53 <MikeSmith> http://w3c-test.org/submissions/
16:31:41 <wseltzer> MikeSmith: Every PR has the state of the PR branch ^
16:31:55 <wseltzer> ... I just tend to check out locally
16:35:38 <hhalpin> RRSAgent, draft minutes
16:35:38 <RRSAgent> I have made the request to generate http://www.w3.org/2016/03/25-crypto-minutes.html hhalpin
16:38:43 <wseltzer> hhalpin: If we get WebIDL, keys, and then figure out service workers with ms2ger
16:41:46 <wseltzer> q+
16:42:49 <wseltzer> q-
16:43:07 <wseltzer> MikeSmith: I recommend that the group and owners set a realistic timeline
16:43:12 <wseltzer> ... that will help me to help you
16:45:15 <wseltzer> rrsagent, make minutes
16:45:15 <RRSAgent> I have made the request to generate http://www.w3.org/2016/03/25-crypto-minutes.html wseltzer
17:13:27 <wseltzer> rrsagent, make logs public
17:13:29 <wseltzer> rrsagent, make minutes
17:13:29 <RRSAgent> I have made the request to generate http://www.w3.org/2016/03/25-crypto-minutes.html wseltzer
17:32:57 <ale> ale has joined #crypto
17:46:29 <Karen> Karen has joined #crypto
19:21:42 <Karen_> Karen_ has joined #crypto
20:57:40 <ale> ale has joined #crypto
21:01:15 <slightlyoff> slightlyoff has joined #crypto
21:01:16 <tobie> tobie has joined #crypto
21:01:37 <timeless> timeless has joined #crypto