See also: IRC log
Mike5: We maintain tests for w3c on github
<Mike5> https://github.com/w3c/web-platform-tests
<ShaneM> https://github.com/w3c/web-platform-tests
Mike5: Our test suites typically
have tests you can do in JS
... goal is automatic testing
... we have a subdirectory in the GitHub repo for payment
request
<Mike5> https://github.com/w3c/web-platform-tests/tree/master/payment-request
Mike5: any time we have a spec that defines interfaces (via WebIDL), we have a test harness
that we can use that makes it easy to take WebIDL blocks, dump in a test file, and use the IDL harness to test whether the browser that we need to test implements the IDL
scribe: the ".https" is due to payment request only works over secure channel
https://github.com/w3c/web-platform-tests/blob/master/payment-request/interfaces.https.html
Mike5: For every requirement in
the spec we need to have corresponding tests
... a single test file might have a number of tests in it
<Zakim> ShaneM, you wanted to talk a little about protocol testing
ShaneM: Part of talking about web
payments information is that it's a protocol
... there are two things you need to do:
a) hit the payment app with well-shaped information as though coming from the browser
b) the payment app needs to hit something and get responses
scribe: we have pretty good
support for these needs through python and python
extensions
... we have good examples for doing this; just need properly
shaped JSOn
(That's not for today; today's problem is the browser API)
scribe: one thing we can do is
ask the browser vendors to implement a fake payment
method
... so we have a guarantee that there's at least one payment
method that's available
... we also want to test the scenario where we know a payment
method is not available
Mike5: here's our biggest
obstacle....
... with many other specs there's nothing special you have to
implement in the test suite to be able to create a web
application that emulates the normal user flow in using the
feature
... but PR API is different in that after you've constructed a
payment request object, you need to get a payment
reasponse
... getting a payment response in the real world requires
interaction with a payment app of some kind
... you need the emulated merchant site to provide information
and then get back a payment response.
... my thinking originally was that we might not be able to do
this solely in JS
... Shane indicated you can do this in JS..
... in the testing infrastructure that we have , we use a
python server based on HTTP built into pythong
... it provides a self-hosted wb server that lets you send
certain headers
... and lets you control http behavior in certain ways
... and lets you run python code on the server side
... so that let's you emulate server-side behavior (thus beyond
JS)
... so it comes down to somebody writing up something
... and with that we can test requirements related to payment
response
the IDL requirements can be easily tested using our IDL harness, but there are other assertions in the specification that we will need to test (e.g., requirements on user agents)
IJ: Any special perms required or just GitHub?
Mike5: You just need a GitHub account and submit PRs
<ShaneM> public-payments-testing is a mailing list we set up for this a while back
Mike5: I can also set people up with direct write access; but we can start with pull request approach
roy: the goal is to test how the
browser functions in this API?
... how does the test suite get ahold of the chrome
runtime?
ShaneM: It may sound
strange...pretend you are just you and want to run some
tests.
... in the web platform test world you bring up your browser
you want to test
... you navigate to a page that defines a test
... and that page cycles the test
Mike5: One way to see the test results is to run browsers locally on your machine as Shane described
<ShaneM> https://github.com/w3c/web-platform-tests
Mike5: we ALSO have some
automation capabilities so that when you check in a patch
... our tools can run the tests run automatically (e.g.,
currently we do that in both firefox and chrome)
... first, the automation runs the test to see whether it
actually executes, but furthermore, we use this mechanisms to
test
flaky tests...so we run the test 10 times in each browser...and we get a report that the test is not deterministing
deterministic
scribe: so this is one way we validate tests
<Mike5> https://w3c-test.org/submissions/4753/payment-request/
scribe: another thing is that
when you create a PR, we have a mechanism that mirrors it to
the web-based instance of the entire repo
... that 4753 is the PR number
... so if you create a PR, it will end up being mirrored, and
you can share the URL with others so they can test the state of
the code from that branch
... and if you push changes to your PR it will update the
mirrored resource
<Zakim> ShaneM, you wanted to ask about running those tests?
ShaneM: Can I point at such a
resource...and have it selected?
... I think runner won't run tests under "Submissions" since
not in the manifest
... it would help people doing development to drive tests
... I put in IRC earlier a link to the runner
... there's an interactive UI that lets you pick tests and run
them
stan: Why do we need a
server-side component?
... e.g., browser-stored card data
ShaneM: You are correct; you don't need server-side to test basic card
[Agreement that server is for third-party payment apps]
[And that's not today's problem]
<Zakim> Mike, you wanted to comment about the runner
<Mike5> https://w3c-test.org/tools/runner/index.html
Mike5: Also, I want to loop back to the runner
"run tests under path"
scribe: that path depends on a
manifest file in the repo
... the reason it won't work with "submissions" is that
submissions is not in the path
... this web-based runner is a convenient thing..it's not
essential
<Mike5> https://w3c-test.org/submissions/4753/payment-request/payment-request-in-iframe.html
scribe: you can run tests any
time from the actual files that are in the directories
... the nice thing about the runner is that you can run
multiple tests in sequence (given a list of URLs)
... and it also produces the results in JSOn
... and we have a reporting tool that will let you produce nice
reports
... I've worked on the runner code myself and one thing to note
is that it's a bit neglected
... if you run into problems with the runner, there are some
known challenges
... the browser vendors don't use this web-based runner...there
is another runner that drives the browser from an external
process
... if the browser crashes that runner can restart the browser,
etc.
... mozilla and chrome use it
IJ: Remind me what the obstacle is in your mind?
Shane: We need a browser that implements PR API and a payment method
Mike: +1
... the only implementation we have of this is in Android
... today you have to start your smart phone and run tests
there...not fun!
... it would be nice if we could get to the point where we have
support in a desktop build to test against
Stan: It feels to me like having
a dummy payment method would help
... testing cards only would be limited
... you would not testing some features....
... I am happy to try to implement a dummy payment method
(WPT = Web Platform Tests infrastructure)
Shane: Useful to have a tool in the WPT environment
Stan: Can we use a WPT server with Android?
Shane: The android platform can
access it, yes.
... Could help to create a demo payment method
... check out the GitHub repo locally and feel free to test
Stan: I am happy to invest some time in that
ShaneM +1
Mike5: +1
... once we have that, then we can get back to writing the
tests
... the work is reading the spec, identifying the testable
requirements (on the browser)
and writing a set of assertions
scribe: the test harness we have is not exotic
IJ: Example?
Stan: Could you show how to glue the python to the js file?
<Mike5> window.top.test(function() {
<Mike5> window.top.assert_throws({name: 'SecurityError'}, function() {
<Mike5> new PaymentRequest([{supportedMethods: ['foo']}], {total: {label: 'label', amount: {currency: 'USD', value: '5.00'}}});
<Mike5> }, 'If the browsing context of the script calling the constructor is not a top-level browsing context, then throw a SecurityError.');
<Mike5> });
[Shane: Web annotation has an example server]
<Mike5> https://github.com/w3c/web-platform-tests/blob/master/payment-request/payment-request-in-iframe.html
Mike5: this is an older test;
need to verify if this is still a requirement of the spec
... so the spec requirement (assertion) is in the test
... if you were to put this in a top-level browser context, it
won't throw. But will throw from an iframe
https://github.com/w3c/web-platform-tests/tree/master/annotation-protocol
Stan: Reading the README...I don't see a way to link a test case to a particular server-side behavior, which would be useful here
ShaneM: There are some special
file name conventions that allow you to invoke things on the
server
... we call it "routing
<Zakim> Mike, you wanted to mention W3C #testing channel
<ShaneM> http://wptserve.readthedocs.org/en/latest/
Mike5: Get help on #testing (irc.w3.org)
<Mike5> documentation in the WPT server is at http://wptserve.readthedocs.io/en/latest/index.html
IJ: Has FB done testing and could we have them?
Roy: I will look into it
Mike5: As ShaneM said, our main
obstacles are not really technical. Mostly time issues
... I do have time to help answer questions
... i can also make time to sit down and work on this
IJ: Can we pick a specific assertion in the spec and try to get Stan up and running with a test next week?
Mike5: We have tests for some WebIDL bits...
https://w3c.github.io/browser-payment-api/
<Mike5> https://www.w3.org/TR/payment-request/#show-method
Mike5: I think "show" is a good place to start
<Mike5> https://www.w3.org/TR/payment-request/#abort-method
===
If this consultation produced no supported method of paying, then reject acceptPromise with a "NotSupportedError" DOMException, and abort this algorithm.
===
Mike5: If you are writing a test while somebody is implementing in parallel, it's sometimes helpful to provide feedback on the spec
<Mike5> https://www.w3.org/TR/payment-request/#complete-method
IJ: Roy is FB implementation available for testing?
Roy: I will look into it
<Zakim> Mike, you wanted to talk about trusted click events for user actions
[Some discussion about UI]
Mike5: This feature exposes a
user interface where user interaction (e.g., click) is
required
... I thought at first that that required a trusted click event
that might mean you can't do in a test
... but apparently this does not require a trusted click and
therefore we can automate a test
... note we don't have web driver integration in the test suite
at this point
ShaneM: I don't think we can
simulate click events interoperabily
... I'd rather rely on web driver when it's available
... so today some of our tests may be only as far as
semi-automated