16:05:59 RRSAgent has joined #wpwg-test 16:05:59 logging to http://www.w3.org/2017/02/07-wpwg-test-irc 16:06:02 Meeting: PR API Test 16:06:05 Chair: Ian 16:06:09 present+ Mike5 16:06:10 present+ 16:06:12 present+ ShaneM 16:06:15 present+ Stan 16:08:18 I should get travel approval for that... 16:09:41 ==== 16:10:15 agenda+ How w3c organizes test suites 16:10:19 agenda+ what we have so far 16:10:27 agenda+ What we need 16:10:35 agenda? 16:10:43 present+ roy 16:10:50 zakim, take up item 1 16:10:50 agendum 1. "How w3c organizes test suites" taken up [from Ian] 16:11:11 Mike5: We maintain tests for w3c on github 16:11:13 https://github.com/w3c/web-platform-tests 16:11:21 https://github.com/w3c/web-platform-tests 16:12:11 Mike5: Our test suites typically have tests you can do in JS 16:12:15 ...goal is automatic testing 16:12:56 q+ to talk a little about protocol testing 16:13:06 ..we have a subdirectory in the GitHub repo for payment request 16:13:06 https://github.com/w3c/web-platform-tests/tree/master/payment-request 16:13:32 ...any time we have a spec that defines interfaces (via WebIDL), we have a test harness 16:13:54 that we can use that makes it easy to take WebIDL blocks, dump in a test file, and use the IDL harness to test whether the browser that we need to test implements the IDL 16:14:23 ...the ".https" is due to payment request only works over secure channel 16:14:35 https://github.com/w3c/web-platform-tests/blob/master/payment-request/interfaces.https.html 16:14:59 Mike5: For every requirement in the spec we need to have corresponding tests 16:15:17 ...a single test file might have a number of tests in it 16:15:43 ack Sh 16:15:43 ShaneM, you wanted to talk a little about protocol testing 16:16:07 ShaneM: Part of talking about web payments information is that it's a protocol 16:16:23 ...there are two things you need to do: 16:16:35 a) hit the payment app with well-shaped information as though coming from the browser 16:16:45 b) the payment app needs to hit something and get responses 16:16:56 ...we have pretty good support for these needs through python and python extensions 16:17:06 ..we have good examples for doing this; just need properly shaped JSOn 16:17:19 (That's not for today; today's problem is the browser API) 16:17:30 ...one thing we can do is ask the browser vendors to implement a fake payment method 16:18:01 ...so we have a guarantee that there's at least one payment method that's available 16:18:18 ...we also want to test the scenario where we know a payment method is not available 16:18:20 ack mik 16:18:49 Mike5: here's our biggest obstacle.... 16:19:11 ...with many other specs there's nothing special you have to implement in the test suite to be able to create a web application that emulates the normal user flow in using the feature 16:19:34 ..but PR API is different in that after you've constructed a payment request object, you need to get a payment reasponse 16:19:57 ...getting a payment response in the real world requires interaction with a payment app of some kind 16:20:12 ...you need the emulated merchant site to provide information and then get back a payment response. 16:20:23 ...my thinking originally was that we might not be able to do this solely in JS 16:20:33 ...Shane indicated you can do this in JS.. 16:20:53 ...in the testing infrastructure that we have , we use a python server based on HTTP built into pythong 16:21:05 ...it provides a self-hosted wb server that lets you send certain headers 16:21:12 ...and lets you control http behavior in certain ways 16:21:23 ..and lets you run python code on the server side 16:21:36 ...so that let's you emulate server-side behavior (thus beyond JS) 16:22:08 ...so it comes down to somebody writing up something 16:22:15 ...and with that we can test requirements related to payment response 16:22:38 the IDL requirements can be easily tested using our IDL harness, but there are other assertions in the specification that we will need to test (e.g., requirements on user agents) 16:23:05 IJ: Any special perms required or just GitHub? 16:23:14 Mike5: You just need a GitHub account and submit PRs 16:23:18 public-payments-testing is a mailing list we set up for this a while back 16:23:29 ...I can also set people up with direct write access; but we can start with pull request approach 16:24:16 roy: the goal is to test how the browser functions in this API? 16:24:25 ...how does the test suite get ahold of the chrome runtime? 16:24:57 ShaneM: It may sound strange...pretend you are just you and want to run some tests. 16:25:11 ...in the web platform test world you bring up your browser you want to test 16:25:18 ...you navigate to a page that defines a test 16:25:23 ..and that page cycles the test 16:26:19 Mike5: One way to see the test results is to run browsers locally on your machine as Shane described 16:26:33 https://github.com/w3c/web-platform-tests 16:26:34 ...we ALSO have some automation capabilities so that when you check in a patch 16:27:00 ...our tools can run the tests run automatically (e.g., currently we do that in both firefox and chrome) 16:27:29 ...first, the automation runs the test to see whether it actually executes, but furthermore, we use this mechanisms to test 16:27:53 flaky tests...so we run the test 10 times in each browser...and we get a report that the test is not deterministing 16:27:57 deterministic 16:28:07 ..so this is one way we validate tests 16:28:27 https://w3c-test.org/submissions/4753/payment-request/ 16:28:29 ...another thing is that when you create a PR, we have a mechanism that mirrors it to the web-based instance of the entire repo 16:28:40 ...that 4753 is the PR number 16:29:05 ...so if you create a PR, it will end up being mirrored, and you can share the URL with others so they can test the state of the code from that branch 16:29:13 q+ to ask about running those tests? 16:29:13 ...and if you push changes to your PR it will update the mirrored resource 16:29:40 ack S 16:29:40 ShaneM, you wanted to ask about running those tests? 16:30:04 ShaneM: Can I point at such a resource...and have it selected? 16:30:14 q+ question about the server requirement 16:30:22 q+ 16:30:46 ShaneM: I think runner won't run tests under "Submissions" since not in the manifest 16:30:58 ...it would help people doing development to drive tests 16:31:11 q+ to comment about the runner 16:31:14 ...I put in IRC earlier a link to the runner 16:31:21 ...there's an interactive UI that lets you pick tests and run them 16:31:29 ack stan 16:31:44 stan: Why do we need a server-side component? 16:31:53 ...e.g., browser-stored card data 16:32:04 ShaneM: You are correct; you don't need server-side to test basic card 16:32:31 [Agreement that server is for third-party payment apps] 16:32:40 [And that's not today's problem] 16:32:55 ack Mi 16:32:55 Mike, you wanted to comment about the runner 16:33:03 https://w3c-test.org/tools/runner/index.html 16:33:06 Mike5: Also, I want to loop back to the runner 16:33:20 "run tests under path" 16:33:26 ..that path depends on a manifest file in the repo 16:33:41 ...the reason it won't work with "submissions" is that submissions is not in the path 16:33:57 ...this web-based runner is a convenient thing..it's not essential 16:34:07 https://w3c-test.org/submissions/4753/payment-request/payment-request-in-iframe.html 16:34:08 ...you can run tests any time from the actual files that are in the directories 16:34:32 ...the nice thing about the runner is that you can run multiple tests in sequence (given a list of URLs) 16:34:38 ..and it also produces the results in JSOn 16:34:47 ..and we have a reporting tool that will let you produce nice reports 16:35:00 ...I've worked on the runner code myself and one thing to note is that it's a bit neglected 16:35:16 ..if you run into problems with the runner, there are some known challenges 16:35:36 ...the browser vendors don't use this web-based runner...there is another runner that drives the browser from an external process 16:35:48 ...if the browser crashes that runner can restart the browser, etc. 16:35:59 ...mozilla and chrome use it 16:36:00 q? 16:36:43 IJ: Remind me what the obstacle is in your mind? 16:37:14 Shane: We need a browser that implements PR API and a payment method 16:37:21 Mike: +1 16:37:26 q+ 16:37:30 ..the only implementation we have of this is in Android 16:37:53 ...today you have to start your smart phone and run tests there...not fun! 16:38:03 ..it would be nice if we could get to the point where we have support in a desktop build to test against 16:38:12 ack s 16:38:41 Stan: It feels to me like having a dummy payment method would help 16:38:48 ...testing cards only would be limited 16:39:02 ...you would not testing some features.... 16:39:28 ...I am happy to try to implement a dummy payment method 16:39:55 (WPT = Web Platform Tests infrastructure) 16:40:09 Shane: Useful to have a tool in the WPT environment 16:40:43 Stan: Can we use a WPT server with Android? 16:40:58 Shane: The android platform can access it, yes. 16:41:18 Shane: Could help to create a demo payment method 16:41:26 ...check out the GitHub repo locally and feel free to test 16:41:33 Stan: I am happy to invest some time in that 16:41:40 ShaneM +1 16:41:42 Mike5: +1 16:41:56 Mike5: once we have that, then we can get back to writing the tests 16:42:10 ...the work is reading the spec, identifying the testable requirements (on the browser) 16:42:15 and writing a set of assertions 16:42:22 ..the test harness we have is not exotic 16:42:36 IJ: Example? 16:42:46 q+ 16:42:54 ack st 16:43:07 Stan: Could you show how to glue the python to the js file? 16:43:42 window.top.test(function() { 16:43:42 window.top.assert_throws({name: 'SecurityError'}, function() { 16:43:42 new PaymentRequest([{supportedMethods: ['foo']}], {total: {label: 'label', amount: {currency: 'USD', value: '5.00'}}}); 16:43:45 }, 'If the browsing context of the script calling the constructor is not a top-level browsing context, then throw a SecurityError.'); 16:43:48 }); 16:43:55 [Shane: Web annotation has an example server] 16:44:22 https://github.com/w3c/web-platform-tests/blob/master/payment-request/payment-request-in-iframe.html 16:44:43 Mike5: this is an older test; need to verify if this is still a requirement of the spec 16:44:56 ...so the spec requirement (assertion) is in the test 16:45:56 ...if you were to put this in a top-level browser context, it won't throw. But will throw from an iframe 16:46:37 https://github.com/w3c/web-platform-tests/tree/master/annotation-protocol 16:47:19 Stan: Reading the README...I don't see a way to link a test case to a particular server-side behavior, which would be useful here 16:47:23 q+ to mention W3C #testing channel 16:47:33 ShaneM: There are some special file name conventions that allow you to invoke things on the server 16:47:37 ...we call it "routing 16:47:48 ack mike 16:47:48 Mike, you wanted to mention W3C #testing channel 16:47:53 http://wptserve.readthedocs.org/en/latest/ 16:48:07 Mike5: Get help on #testing (irc.w3.org) 16:49:04 I have made the request to generate http://www.w3.org/2017/02/07-wpwg-test-minutes.html Ian 16:49:31 RRSAgent, set logs public 16:49:43 documentation in the WPT server is at http://wptserve.readthedocs.io/en/latest/index.html 16:50:28 IJ: Has FB done testing and could we have them? 16:50:32 Roy: I will look into it 16:51:01 Mike5: As ShaneM said, our main obstacles are not really technical. Mostly time issues 16:51:07 ..I do have time to help answer questions 16:51:49 ...i can also make time to sit down and work on this 16:53:01 IJ: Can we pick a specific assertion in the spec and try to get Stan up and running with a test next week? 16:54:09 Mike5: We have tests for some WebIDL bits... 16:54:20 https://w3c.github.io/browser-payment-api/ 16:54:22 https://www.w3.org/TR/payment-request/#show-method 16:54:31 Mike5: I think "show" is a good place to start 16:54:44 https://www.w3.org/TR/payment-request/#abort-method 16:54:53 === 16:54:55 If this consultation produced no supported method of paying, then reject acceptPromise with a "NotSupportedError" DOMException, and abort this algorithm. 16:54:59 === 16:55:24 Mike5: If you are writing a test while somebody is implementing in parallel, it's sometimes helpful to provide feedback on the spec 16:55:55 https://www.w3.org/TR/payment-request/#complete-method 16:57:19 q+ about trusted click events for user actions 16:57:20 IJ: Roy is FB implementation available for testing? 16:57:24 q+ to talk about trusted click events for user actions 16:57:25 Roy: I will look into it 16:57:27 q? 16:57:30 ack Mike 16:57:30 Mike, you wanted to talk about trusted click events for user actions 16:57:45 [Some discussion about UI] 16:57:59 Mike5: This feature exposes a user interface where user interaction (e.g., click) is required 16:58:17 ...I thought at first that that required a trusted click event that might mean you can't do in a test 16:58:36 ...but apparently this does not require a trusted click and therefore we can automate a test 16:58:54 ....note we don't have web driver integration in the test suite at this point 16:59:09 ShaneM: I don't think we can simulate click events interoperabily 16:59:17 ...I'd rather rely on web driver when it's available 16:59:26 ...so today some of our tests may be only as far as semi-automated 16:59:47 RRSAgent, make minutes 16:59:47 I have made the request to generate http://www.w3.org/2017/02/07-wpwg-test-minutes.html Ian