10:57:42 RRSAgent has joined #auto 10:57:42 logging to http://www.w3.org/2017/03/16-auto-irc 10:57:44 RRSAgent, make logs public 10:57:44 Zakim has joined #auto 10:57:46 Zakim, this will be 10:57:46 I don't understand 'this will be', trackbot 10:57:47 Meeting: Automotive Working Group Teleconference 10:57:47 Date: 16 March 2017 10:59:00 urata_access has joined #auto 11:02:38 https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco 11:03:32 Meeting: Testing 11:03:35 scribenick: ted 11:03:40 Scribe: Ted 11:03:46 Chair: Urata 11:04:02 peterw has joined #auto 11:04:13 mike has joined #auto 11:05:33 Present+ Urata, Peter, Ted, Junichi, Hira, Mike, Wonsuk 11:06:00 Urata reviews agenda 11:06:13 Agenda: https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco 11:07:12 Topic: Testing Document 11:07:37 -> https://github.com/w3c/automotive/issues/123 Issue 123 11:08:31 ted: I reviewed this after last call. it is a very nice presentation, much of it is from W3C's Process Document 11:09:02 … we should follow W3C's Process Document and not create a new process 11:09:15 hira reviews slidedeck for call 11:10:16 [there are great examples in slides from W3C MMI activity http://www.w3.org/2002/mmi/] 11:12:29 [slide 5 Implementation report objectives] 11:14:20 peter: is this implementation report we as a group produce or something the different implementers? 11:15:00 ted: ideally the implementers since they are more familiar with their own implementation 11:15:25 … if they do not and are not part of the group, we can. we don't have to write reports for all implementations 11:15:49 hira: w3c requires two implementations as proof of interoperability 11:16:35 ted: to enter CR we need a plan for creating implementation reports and complete them during CR phase 11:16:50 [slide 6] 11:19:49 -> https://www.w3.org/2015/Process-20150901/#implementation-experience Process Document on Implementation experiences (report) 11:24:04 [slide 7] 11:27:27 Urata asks about VIAS entering CR 11:27:55 [discussion of loss of editor, needs wider group review and be published as FPWD] 11:31:46 [peer pressure by group on Urata to step up. best would be to find two editors] 11:31:56 Urata: I will consider and discuss with my supervisor 11:32:10 Present+ Adam 11:33:21 Wonsuk: would we need browser implementations for VIAS as that would be a problem? 11:33:45 Ted: we need two implementations, they do not have to be in a browser 11:36:29 Hira: (from slide 7) signal generator out of scope 11:37:20 Ted: agree but people will need a source, record and replay, generator or actual vehicle 11:37:28 Peter: we use Open VS 11:37:41 https://www.opends.eu/ 11:37:48 AdamC has joined #auto 11:37:54 s/we use Open VS/we use Open DS/ 11:39:15 Hira: we have a list of 30-40 assertions that can be tested and store them in a spreadsheet 11:39:41 … please review the list yourselves 11:39:58 Urata: I am difficulty implementing some tests 11:40:13 … authorization system is not clear 11:40:39 … access control state is implementation dependent concept and not defined clearly in our spec 11:40:54 … it is hard to realize in a prototype and create test for it 11:41:15 … a mock security provider can be used with predefined fake tokens 11:41:42 … this is one possible way I am considering 11:41:59 … I would like to hear from others 11:42:39 ahaller2 has joined #auto 11:43:22 sam has joined #auto 11:45:14 Ted: I think it is a workable idea and would suggest instead of 'aaaaaa' 'ddddd' using strings that explain the token type 11:45:42 … eg 'this_is_full_access_token' 'this_is_invalid_token' and 'this_is_limited_access_token' 11:46:32 wonsuk has joined #auto 11:46:38 I have made the request to generate http://www.w3.org/2017/03/16-auto-minutes.html ted 11:48:45 Hira: what issues remain? 11:48:54 Urata: I have a report in github issues 11:49:31 [slide 12] 11:50:18 Hira: what do you think of the diagram? 11:50:30 Peter: that architecture is similar but we will use real signals instead of a generator 11:50:47 … we will not have a full tree but will be limited by what the vehicle makes available 11:51:11 … we will be able to test with different gets/sets 11:51:20 petew has joined #auto 11:52:39 Urata: a signal generator can create all signals. they might not be as realistic as from a production vehicle but still useful 11:53:26 Hira: how does your generator connect to VISS server? 11:53:34 Urata: I used web sockets 11:54:19 [slide 14] 11:57:52 Ted: I am confused by the diagram in slide 14. We are not doing anything with Geolocation, not sure what is meant by Web Storage, we do not do HTTP at present so wonder why there are multiple HTTP servers 11:58:22 … we need to test web socket. that can but doesn't have to be done from a browser using a test harness 11:59:05 Hira: Web storage and geolocation not needed 11:59:54 Urata: I created this picture to explain 12:02:37 [Urata broadcasting in WebEx editing diagram, clarifying that the tests will be served up by http] 12:03:50 [html ui for testing. loads JS harness, tests and then makes ws connection for json] 12:03:55 kaz has joined #auto 12:04:08 Urata: Geolocation was an example of mixing in other API 12:08:08 Wonsuk: for the signal generator, can we specify our own data for testing? 12:08:40 Urata: that is a problem. I have to use specific signals, engine speed for example, in my tests 12:09:27 Wonsuk: we need to test for different data types, integer, string... their size limits 12:10:12 Urata: (highlighting a specific test in spreadsheet) we also set min/max values 12:12:42 Ted: what might be useful since people will have different data sets as mentioned is to have a generic test for checking a signal and then set what they want to test in configuration file 12:13:02 … which signal, expected type and range of legitimate values 12:13:17 … also tokens can be configuration parameters 12:13:39 … it would be good to have defaults - signals to expect and test tokens 12:13:52 Urata: that is one possible way 12:16:25 Ted: this way your test suite can be used to check production environments with what signals the underlying vehilce provides (and does not allow access) 12:19:14 Sorry to step away for a second, and will be back soon. 12:19:45 Wonsuk: client test cases should start with getVSS and based on the response the client knows what sort of signals it can test 12:20:12 … then you can have dynamic tests 12:22:33 Ted: +1 that would be very useful but might be more complicated to implement 12:22:44 Urata: that would be very good but not sure I have enough time 12:23:22 … if getVSS method fails then it fails the whole test 12:23:42 … configuration based is reasonable to do 12:28:17 Peter: agree you want to be able to specify signals to test and would ideally want to run getVSS and then test each signal 12:28:17 Ted: if that is too complicated then individual tester could start with getVSS and use the response and send through a configuration generator so a subsequent run will test all those signals 12:30:36 … also remember VSS can be extended with custom signals and it would be nice for people to be able to test those as well 12:32:03 Topic: Introduce Test Suite 12:32:48 https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco 12:34:51 https://github.com/aShinjiroUrata/vehicle-signal-server-spec 12:34:57 -> https://lists.w3.org/Archives/Member/member-automotive/2017Mar/att-0047/170310_testsuite_cap2.png Test result 12:35:48 s|https://github.com/aShinjiroUrata/vehicle-signal-server-spec|-> https://github.com/aShinjiroUrata/vehicle-signal-server-spec Test suite repository| 12:36:08 Urata: get, set, subscribe, getVSS all supported 12:36:37 … not supported is filter and @@s 12:37:25 … wss (secure web sockets) is not supported 12:38:49 Ted: problem is the certificates right? 12:38:53 Urata: yes 12:39:58 Ted: in many programming languages you can tell either in the connection method or as part of the platform configuration you can tell your code not to validate certificates 12:40:16 Peter: self signing is much easier 12:40:29 … it is useful for us to test secure sockets 12:44:29 http://163.44.169.166:8081/test-ui.html 12:48:43 s|http://163.44.169.166:8081/test-ui.html|| 12:49:24 … you could also allow tester to upload/provide a cert and store under /etc/ssl (or better somewhere under /tmp/ssl and have your OS set to check there before trying to validate online) but that is more complicated. easiest is if your code can simply not check cert 12:49:56 Urata presenting html UI running in his browser. it is temporarily available on an IP URI shared with the group in IRC but will not be in minutes 12:50:36 Urata: through the UI I can get or subscribe to just about anything 13:03:16 … I need to revise the README so others can install 13:05:44 https://github.com/aShinjiroUrata/vehicle-signal-server-spec 13:07:09 https://github.com/w3c/automotive/viss-test-suite 13:07:45 Urata: do I need to have test suite in W3C repo, when, process for review 13:08:19 Ted: it doesn't have to be but agree better. please do initial import and that way people can review and handle merge requests for new tests 13:10:50 … it is clearer others can use and contribute by having it in w3c repo. you can continue with your fork as well 13:11:46 Urata: ok to do in April? 13:11:52 Ted: yes, when you're ready 13:12:09 Urata: what should we do with VIAS test suite? 13:19:52 Ted: as you say, FPWD after we get editors 13:24:22 … we are far off on our timeline. also we have been reaching out and trying to engage PSA+IBM, Visteon (who applied but hasn't joined yet) and GM who all have competing efforts to VIAS that are all further along 13:25:15 … we should continue with VIAS like VISS because we cannot be sure of their participation. VISS is looking good for schedule, VIAS not 13:28:50 https://github.com/aShinjiroUrata/web-platform-tests/tree/dev-urata-vsss-test/vehicle-signal-server-spec 13:30:46 s|https://github.com/aShinjiroUrata/web-platform-tests/tree/dev-urata-vsss-test/vehicle-signal-server-spec|-> https://github.com/aShinjiroUrata/web-platform-tests/tree/dev-urata-vsss-test/vehicle-signal-server-spec Test cases for VISS| 13:31:07 s|https://github.com/aShinjiroUrata/vehicle-signal-server-spec|-> https://github.com/aShinjiroUrata/vehicle-signal-server-spec Test Framework for VISS| 13:31:31 Wonsuk: when do you expect you can make this available for use? 13:31:46 … I have a need to use these myself 13:32:11 Urata: you can clone the repo and use it now 13:32:22 … you will need to install on a linux machine 13:34:18 [in WebEx chat Mike indicated he can contribute to JS & Node, can contribute to testing and Junichi tries to recruit him as second VIAS editor. junichi-hashimoto++ ! ] 13:35:07 Hira: when do you expect the remaining test cases you indicated to be available? 13:35:15 Urata: hopefully by next week 13:36:29 Ted: that would be great but thinking improving README so others can run and write tests might be higher priority 13:36:37 … people may think of additional useful tests 13:36:42 Peter: we're interested 13:39:03 Wonsuk: browser may block self signed certificates, there are some tricks when launching browser I can help with in Mac 13:39:27 Junichi: on Mac you can open login keychain that Chromium would accept 13:40:03 … we have to use self signed because we are not using a fully qualified domain name that a certificate authority could sign 13:40:38 browser launching option I use: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/tmp/demo/ --ignore-certificate-errors & 13:41:24 Urata: local server can have self signed 13:41:39 Junichi: in our case it isn't possible since it is a special name 13:42:30 I have made the request to generate http://www.w3.org/2017/03/16-auto-minutes.html ted 13:44:44 ahaller2 has joined #auto 13:46:35 Urata: are there any other topics that people want to discuss? 13:47:15 [adjourned] 13:50:24 I have made the request to generate http://www.w3.org/2017/03/16-auto-minutes.html ted 13:53:30 I see no action items