Automotive Working Group Teleconference on Testing

<urata_access> https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco

Urata reviews agenda

Testing Document

Issue 123

ted: I reviewed this after last call. it is a very nice presentation, much of it is from W3C's Process Document
... we should follow W3C's Process Document and not create a new process

Hira reviews slidedeck for call

[there are great examples in slides from W3C MMI activity http://www.w3.org/2002/mmi/]

[slide 5 Implementation report objectives]

Peter: is this implementation report we as a group produce or something the different implementers?

Ted: ideally the implementers since they are more familiar with their own implementation
... if they do not and are not part of the group, we can. we don't have to write reports for all implementations

Hira: w3c requires two implementations as proof of interoperability

Ted: to enter CR we need a plan for creating implementation reports and complete them during CR phase

[slide 6]

Process Document on Implementation experiences (report)

[slide 7]

Urata asks about VIAS entering CR

[discussion of loss of editor, needs wider group review and be published as FPWD]

[peer pressure by group on Urata to step up. best would be to find two editors]

Urata: I will consider and discuss with my supervisor

Wonsuk: would we need browser implementations for VIAS as that would be a problem?

Ted: we need two implementations, they do not have to be in a browser

Hira: (from slide 7) signal generator out of scope

Ted: agree but people will need a source, record and replay, generator or actual vehicle

Peter: we use Open DS

<urata_access> https://www.opends.eu/

Hira: we have a list of 30-40 assertions that can be tested and store them in a spreadsheet (see archived copy)
... please review the list yourselves

Urata: I am difficulty implementing some tests
... authorization system is not clear
... access control state is implementation dependent concept and not defined clearly in our spec
... it is hard to realize in a prototype and create test for it
... a mock security provider can be used with predefined fake tokens
... this is one possible way I am considering
... I would like to hear from others

Ted: I think it is a workable idea and would suggest instead of 'aaaaaa' 'ddddd' using strings that explain the token type
... eg 'this_is_full_access_token' 'this_is_invalid_token' and 'this_is_limited_access_token'

Hira: what issues remain?

Urata: I will report in github issues

[slide 12]

Hira: what do you think of the diagram?

Peter: that architecture is similar but we will use real signals instead of a generator
... we will not have a full tree but will be limited by what the vehicle makes available
... we will be able to test with different gets/sets

Urata: a signal generator can create all signals. they might not be as realistic as from a production vehicle but still useful

Hira: how does your generator connect to VISS server?

Urata: I used web sockets

[slide 14]

Ted: I am confused by the diagram in slide 14. We are not doing anything with Geolocation, not sure what is meant by Web Storage, we do not do HTTP at present so wonder why there are multiple HTTP servers
... we need to test web socket. that can but doesn't have to be done from a browser using a test harness

Hira: Web storage and geolocation not needed

Urata: I created this picture to explain

[Urata broadcasting in WebEx editing diagram, clarifying that the tests will be served up by http]

[html ui for testing. loads JS harness, tests and then makes ws connection for json]

Urata: Geolocation was an example of mixing in other API

Wonsuk: for the signal generator, can we specify our own data for testing?

Urata: that is a problem. I have to use specific signals, engine speed for example, in my tests

Wonsuk: we need to test for different data types, integer, string... their size limits

Urata: (highlighting a specific test in spreadsheet) we also set min/max values

Ted: what might be useful since people will have different data sets as mentioned is to have a generic test for checking a signal and then set what they want to test in configuration file
... which signal, expected type and range of legitimate values
... also tokens can be configuration parameters
... it would be good to have defaults - signals to expect and test tokens

Urata: that is one possible way

Ted: this way your test suite can be used to check production environments with what signals the underlying vehilce provides (and does not allow access)

<hira> Sorry to step away for a second, and will be back soon.

Wonsuk: client test cases should start with getVSS and based on the response the client knows what sort of signals it can test
... then you can have dynamic tests

Ted: +1 that would be very useful but might be more complicated to implement

Urata: that would be very good but not sure I have enough time
... if getVSS method fails then it fails the whole test
... configuration based is reasonable to do

Peter: agree you want to be able to specify signals to test and would ideally want to run getVSS and then test each signal

Ted: if that is too complicated then individual tester could start with getVSS and use the response and send through a configuration generator so a subsequent run will test all those signals
... also remember VSS can be extended with custom signals and it would be nice for people to be able to test those as well

Introduce Test Suite

<hira> https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco

<urata_access> Test suite repository

Test result

Urata: wrote support status in README of github. get, set, subscribe, getVSS all supported
... not supported is filter and wildcards
... wss (secure web sockets) is not supported

Ted: problem is the certificates right?

Urata: yes

Ted: in many programming languages you can tell either in the connection method or as part of the platform configuration you can tell your code not to validate certificates

Peter: self signing is much easier
... it is useful for us to test secure sockets
... you could also allow tester to upload/provide a cert and store under /etc/ssl (or better somewhere under /tmp/ssl and have your OS set to check there before trying to validate online) but that is more complicated. easiest is if your code can simply not check cert

Urata presenting html UI running in his browser. it is temporarily available on an IP URI shared with the group in IRC but will not be in minutes

Urata: through the UI I can get or subscribe to just about anything
... I need to revise the README so others can install

<urata_access> Test Framework for VISS

https://github.com/w3c/automotive/viss-test-suite

Urata: do I need to have test suite in W3C repo, when, process for review

Ted: it doesn't have to be but agree better. please do initial import and that way people can review and handle merge requests for new tests
... it is clearer others can use and contribute by having it in w3c repo. you can continue with your fork as well

Urata: ok to do in April?

Ted: yes, when you're ready

Urata: what should we do with VIAS test suite?

Ted: as you say, FPWD after we get editors
... we are far off on our timeline. also we have been reaching out and trying to engage PSA+IBM, Visteon (who applied but hasn't joined yet) and GM who all have competing efforts to VIAS that are all further along
... we should continue with VIAS like VISS because we cannot be sure of their participation. VISS is looking good for schedule, VIAS not

<urata_access> Test cases for VISS

Wonsuk: when do you expect you can make this available for use?
... I have a need to use these myself

Urata: you can clone the repo and use it now
... you will need to install on a linux machine

[in WebEx chat Mike indicated he can contribute to JS & Node, can contribute to testing and Junichi tries to recruit him as second VIAS editor. junichi-hashimoto++ ! ]

Hira: when do you expect the remaining test cases you indicated to be available?

Urata: hopefully by next week

Ted: that would be great but thinking improving README so others can run and write tests might be higher priority
... people may think of additional useful tests

Peter: we're interested

Wonsuk: browser may block self signed certificates, there are some tricks when launching browser I can help with in Mac

Junichi: on Mac you can open login keychain that Chromium would accept
... we have to use self signed because we are not using a fully qualified domain name that a certificate authority could sign

<wonsuk> browser launching option I use: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/tmp/demo/ --ignore-certificate-errors &

Urata: local server can have self signed

Junichi: in our case it isn't possible since it is a special name

Urata: are there any other topics that people want to discuss?

[adjourned]