15:25:33 RRSAgent has joined #did-topic 15:25:33 logging to https://www.w3.org/2021/04/01-did-topic-irc 15:25:35 RRSAgent, make logs Public 15:25:36 please title this meeting ("meeting: ..."), ivan 15:26:01 Meeting: DID WG Topic Call on the Test Suite 15:26:01 Chair: brent 15:26:01 Date: 2021-04-01 15:26:01 Agenda: https://lists.w3.org/Archives/Public/public-did-wg/2021Mar/0026.html 16:01:26 present+ 16:02:11 Orie has joined #did-topic 16:02:11 present+ 16:02:20 present+ orie 16:02:28 present+ markus 16:02:36 markus_sabadello has joined #did-topic 16:03:07 scribe+ rhiaro 16:04:01 https://lists.w3.org/Archives/Public/public-did-wg/2021Mar/0026.html 16:06:33 all: meditative silence 16:07:35 Orie: I'd like to be able to merge PRs but I'm hoping other folks will be able to merge PRs. We need more than one person to be able to merge a PR 16:07:41 ... to progress at a reasonable pace 16:07:51 ... I want others to feel empowered to do that 16:07:55 markus_sabadello: it's you and manu right now 16:08:00 Orie: that's problematic. We need more 16:08:29 ... agree not enough of us on the call to do much, other than merge some of your PRs if they've been open for some time 16:08:33 markus_sabadello: 6 days, but only one review 16:09:05 Orie: we need more review 16:09:07 present+ 16:09:17 markus_sabadello: biggest challenge is overlap between work that different people are doing 16:09:28 ... eg. at least three people ahve implemented a test to see if a prperty value conforms to an xml datetime 16:09:34 ... beause it's a requirement that comes up in several parts of the spec 16:09:42 ... so people implement that separately, so duplication in the code 16:09:45 ... and references between sections 16:09:57 ... your view was this isn't necessarily a bad thing and people should work in paralell and harmonize later 16:10:03 Orie: people should own their sectin, then we should do cleanup, and repeat 16:10:08 ... not worry about code duplication immediately 16:10:18 ... at some point a tremendous amount of helper functions that should be centralised and reused 16:10:26 ... check url, did url, ascii, datetime, etc 16:10:58 Orie: which PR should be merged markus_sabadello? 42? 16:11:23 markus_sabadello: I'd rather have 43 merged. A lot of code on testing the did resolution and url dereferencing sections 16:11:26 ... a lot of stuff 16:11:29 ... to avoid future conflicts 16:11:39 ... and to start realising what kind of structural change we might want 16:11:42 Orie: I'm merging it. 16:11:46 ... Feel the pain f not doing reviews. 16:11:52 ... clean it up later.. 16:12:02 https://github.com/w3c/did-test-suite/pull/44 16:12:08 markus_sabadello: charles has 2 open PRs, I reviewed one 16:12:15 Orie: let's look at 44 together. i reviwed 16:12:19 ... Organise DID param tests by section 16:12:34 ... section annotation prefixing, so we can tell what sections are tested, looks good 16:12:39 ... thrust of this PR, charles? 16:12:40 cel: yes 16:12:45 ... in response to a comment by manu on the previous PR 16:12:55 Orie: cool. it's been open for 5 days. Any reviews before I just merge it? 16:12:58 ... can I merge it? 16:13:00 markus_sabadello: approved 16:13:03 Orie: merged 16:13:13 ... PR 41... not looking at that 16:13:16 https://github.com/w3c/did-test-suite/pull/45 16:13:28 ... PR 45.. test conforming consumer 16:13:43 ... this is asserting that errors are produced for different representations that when consumed might result in an error? 16:13:47 cel: that's part of it 16:14:08 Orie: sections are annotated well, expectations for errors are basically reflected through the fixture, nice DID regex there 16:14:14 markus_sabadello: this is the kind of thing we probably want to reuse 16:14:19 ... I also have tests that test if something is a valid DID 16:14:24 ... and a regex but it's less cool than this one 16:14:33 Orie: I do too... there's a million.. they're all broken because of the % encoding 16:14:42 ... % encoding is included in this one. Great. 16:14:59 markus_sabadello: I like about the input data that the config.json with the data is the errors. If I understand correctly it allows negative tests 16:15:13 ... inputs that are not valid and test if the result of the consumption in this case is an error as expected 16:15:17 Orie: looks great 16:15:19 markus_sabadello: I tried to do that too 16:15:21 ... in resolution 16:15:36 ... we may want to reuse some conventions on how we express negative outcomes 16:15:37 Orie: agreed 16:15:45 ... any objection to merging 45? it's been open for 3 days and has 2 approvals 16:15:49 ... merging 16:16:07 ... shigeya, missing dependency... non spec related... going to merge it 16:16:11 ... just trusting him 16:16:17 https://github.com/w3c/did-test-suite/pull/47 16:16:17 ... now his most recent PR 47 16:16:26 ... is about metadata structures. I started a review 16:16:32 ... we might ask him to include the section headers in the descriptions 16:16:38 ... but he does have one section header for metadata structure 16:16:43 ... I didn't get all the way through reviewing 16:16:45 ... this may be sufficient 16:17:05 ... Looks good. Approving. 16:17:24 markus_sabadello: I worked in a different suite 16:17:31 brent has joined #did-topic 16:17:37 Orie: that's really helpful at this stage to have separate files and directories, smaller files are always safer 16:17:39 present+ 16:17:48 ... I propose everybody on the call take a look at 47, call out anything problematic 16:18:00 ... it's short so you should be able to request changes or approve it in the next couple of minutes 16:19:07 everyone: sounds of concentration as PR is carefully reviewed 16:19:37 markus_sabadello: one test that can't easily be tested in code. I also had some of those in my section. He made it a ?? which is also fine 16:19:44 Orie: it's really good, it'll show up as a thing that we'll have to look at 16:19:48 ... if we can't get rid of it we'll remove it 16:20:03 ... if this were automatic our lives would be totally different 16:20:12 ... it's beyond the reasonale expectation of a test author 16:20:38 ... any objection to merging 47? 16:20:52 markus_sabadello: I think it's fine 16:20:59 Orie: we can always open a PR to update it in the future 16:21:05 ... we know these tests are going to be changed around a lot 16:21:15 ... no merge conflicts, 3 approvals, going ot merge 16:21:20 ... now we're talking about the production tests 16:21:32 ... markus had a changeset requested on this originally because the language was stale, we were testing statements from a pre CR version 16:21:41 ... manu claims to have resolved that, you have change suggestsions, he said he implemented some 16:21:52 ... wondering if we can resole the outdated conversation so folks aren't tripping up over it 16:22:13 ... under your review, where he says he fixed the requested change 16:22:17 ... resolve if that's true 16:22:21 markus_sabadello: definitely resolved 16:22:49 ... in general this PR has a very insufficient.. the language here in the spec talks about the entries in the data model, the representation specific entries 16:22:53 ... and using them both to produce th erpresentaiton 16:23:09 ... and charles did that in a good way in his PR, the one of the two we just merged on conforming consumers 16:23:15 ... in the data he distinguished between the ADM and representations 16:23:19 ... in manu's PR it doesn't happen at all 16:23:23 ... its' doing something but not what the spec says 16:23:28 ... I wonder that's not a reason to not merge it 16:23:32 ... could say it's better to have something than nthing 16:23:35 Orie: we can open an issue for that 16:23:38 ... and we should 16:24:02 ... if we think this test is better than nothing but flawed we should open an issue noting its a problem, describe changes we'd like to see, contineu that conversaiton and the issue will move along and we'll do a PR to address those concerns 16:24:16 ... seems like looking at the suite config here.. it's a little.. interesting structure 16:24:28 ... *reads code* 16:24:40 ... markus, you think the primary issue with this is reliance on did resolution result? 16:24:46 ... it should instead be relying on some ADM representation? 16:24:48 markus_sabadello: yes 16:25:06 ... if you look at what charles did 16:25:25 Orie: I commented on the generate did producer tets line which is the beginning of hte problem 16:25:33 ... if you can link to the part of charles' test that is excellent you should do that 16:26:05 markus_sabadello: I will post the comment, just need a minute 16:26:27 Orie: we don't have any merge conflicts.. if I were to open an issue.. 16:26:35 ... producer tests should be of the ADM 16:26:38 markus_sabadello: yep 16:27:05 ... posted comment 16:27:30 Orie: let's see if this issue if resolved would address your concerns and if you'd be willing to approve 16:28:33 Orie: manu is asserting that the resolution result is the ADM 16:28:54 ... with that issue opened acknowledging it has to be addressed for these tests to meet our needs would you be willing to re review 16:28:58 markus_sabadello: I'd be fine with that 16:29:17 ... wondering if manu is already working on something. His last comment was he'll take a look. Maybe he doesn't want us to merge it at this point becuase he's improving it 16:29:20 ... but ot merge now and improve later is fine 16:29:28 Orie: if he doesn't want it merged he should close the PR. It's open 11 days 16:29:33 ... we've gone around, there's no merge conflict 16:29:42 ... a smaller PR which addresses these concerns would be easier to review 16:29:47 ... in favour of merging and future fixes in a smaller PR 16:29:48 markus_sabadello: fine 16:30:03 Orie: do we think he's coming...? 16:30:15 brent: he got double booked 16:31:18 Orie: perhaps while we wait to hear back from manu we should discuss the next primary roadblock. Markus, what are you looking at? Are you blocked 16:31:26 ... and same for charles 16:31:40 markus_sabadello: we should answer the question if we want one test suite or multiple suites 16:31:45 ... if multiple then how do we split them up 16:31:54 ... do we have a different suite for each section rather than having multiple files in a single test suite? 16:31:57 ... that would be helpful 16:32:06 ... we're far enough along that we could actually try to harmonize the different input files 16:32:11 ... the default test suite there's a lot of duplication 16:32:15 ... a lot of did documents 16:32:28 ... one input file is used for testing production and consumption and a different test suite fro resolution which has a lot of did documents 16:32:40 ... from a perspective of a did method implementer right now they'd have to submit a lot of unnecessarily duplicated data 16:32:53 ... if you have a did method and you want to submit five did docs some with or without errors, right now that's spread out across too many files 16:33:00 ... we could start having this discussion now we merged a lot of PRs 16:33:01 https://github.com/w3c/did-test-suite/issues/49 16:33:02 Orie: agreed 16:33:05 ... I opened issue 49 16:33:14 ... which is proposal to restructure the json to be js files that can include smaller json files 16:33:19 ... helping us not repeat ourselves 16:33:25 .... that's one piece of cleanup we could do 16:33:36 ... regarding the suites, I'm in favour of a suite per section for now 16:33:41 ... assuming we get the configuration pieces sorted out 16:33:59 ... it's good to have smaller test suites more independant, different folders, will lead to less merge conflicts and more autonomy and velocity 16:34:08 ... we should make a proposal to manage multiple suites 16:34:20 ... I'll create an issue for that 16:35:21 https://github.com/w3c/did-test-suite/issues/50 16:35:49 ... I think it will allow us ot test better in isolation, own yoru section of the spec more easily and make it easier to work on tests, testing only the things you care to improve 16:35:58 markus_sabadello: sounds good. Also interested in charles' opinion 16:36:22 cel: either could work 16:36:54 markus_sabadello: we mean a different directory in the repo. You submitted yours in the did spec path where all the others are so far, the proposal is to split that out into folders 16:36:57 cel: I think that'd be good 16:37:10 Orie: related issue is we've beent alking about testing the spec, but ther's a desire to test specific did methods 16:37:15 ... a case where we want separate tests 16:37:34 ... i know manu talked about the desire to have the tests for the spec could be reused for the did method 16:37:39 ... concerned that's a dream and hard in realtiy 16:37:47 ... easier to implement did method conformance tests as an isolated suite 16:37:50 ... anyone have opinions on that? 16:38:02 cel: sounds good to me because this test suite is focussed on testing the normative statements 16:38:06 Orie: yep 16:38:17 markus_sabadello: not sure I understand, was thinking the opposite 16:38:42 ... I thought to test the spec, what we're doing right now, example data and testing the normative statements. In the future people submit their did method results then I didnt think we'd have to write additional code or tests, just additional input data that we add 16:38:48 ... rather than being fictional example data it's real data 16:38:55 ... but other than that we wouldn't do anythign different 16:39:16 Orie: as an implementer wishing to show your method conforms to the 15 different sections of the spec you'd craft payloads that replaced exmaple.com with your did method if youw anted to show cnformance with that section 16:39:28 ... and then we'd have a directory of different configurations for each method across each of the sections of the spec 16:39:32 ... how you imagine it? 16:39:34 markus_sabadello: i think so yeah 16:39:41 Orie: I think that's also how manu imagines it 16:39:57 ... concerned that the experience for actually doing that for a real did method might be more frustrating than having an indpeendant test suite that reuses pieces of the tests for the spec 16:39:59 ... but I can see both ways 16:40:20 cel: either could work but there would be mor changes needed here to test the did methods fully because currently some tests are only triggered if conditions are met 16:40:26 ... and the input might not meet those conditions 16:40:29 Orie: yeah 16:40:41 ... that's part of... generally with tests when you try to make a test do more than one thing you end up sad 16:40:56 ... an opportunity to make a really great test experience for did methods that want to show conformance 16:41:01 ... by giving them a more structured input format 16:41:06 ... we can reuse a lot of code to support them 16:41:18 ... but asking them to submit input matching the same structure of the inputs to test normative statements is potentially burdensome 16:41:28 ... maybe we're not far enough along on the tests to answer whether that's true or not 16:41:32 markus_sabadello: talking bout two repos? 16:41:33 Orie: no 16:41:54 ... like you had created a separate suite for a section of the did core spec. We might create a suite for did method conformance that would call into this different sections fo ra structured input document 16:42:04 ... separating th econcerns of normative statement tests vs make it easy to make a did method to conform to the spec 16:42:09 ... two separate testing goals 16:42:22 ... we know we can force the did methods to produce input fixtures that look like what we have today 16:42:43 ... that might be frustrating for folks, they dont' care about a good number of the normative statements, they just want to provide an example of a did doc after resolution for their did method 16:42:47 ... provide dids with url parameters 16:43:02 ... they migth e hoping for an easier st of input fixtures than what we provide if we require them to submit json fixtures for a bunch of different suites 16:43:14 markus_sabadello: the fixtures we have no have to be harmonized anyway, they will become simpler than they are now 16:43:19 ... the ask of the did method implementers will not be that difficult 16:43:33 ... on the other hand with the special test suite for did method conformance sounds nice. Don't have a strong opinion 16:43:39 ... let's clean up what we have now then we'll see 16:43:42 Orie: that makes sense, agree 16:44:00 ... We only ahve one PR open, no objections from other editors for merging the producer PR 16:44:13 ... two approvals, open for 11 days, issue open to track concerns 16:44:17 ... gonna merge it 16:44:25 ... Now we have no work to do \o/ 16:44:34 cel: april fools 16:44:49 Orie: I asked markus what was blocking, you gave some feedback, opened issues to track yoru feedback 16:45:00 ... charles, do you have any direct feedback? what's blocking you? 16:45:04 ... any things you'd like to discuss? 16:45:12 cel: I'm not sure, thanks. I'm writing my feedback on the new issue 16:45:19 ... and I'm open to taking on some part of the work 16:45:24 Orie: awesome 16:45:31 ... declare yourself as taking on that work on the issue 16:45:40 cel: in general, but I'm writing feedback on issue 48 16:45:53 Orie: that's good. Feels like we might want to make manu address 48 16:45:57 ... it was his PR 16:46:01 ... want to make sure he felt comfortable with it 16:46:07 ... but if you're willing to do the work I'm happy to assign you 16:46:15 ... leave a comment, he'll be happy to review a PR instead of make one 16:46:17 cel: okay 16:47:19 various: github misbehaves 16:47:36 Orie: other unassigned issues 16:47:40 ... proposal to refactor input json 16:47:47 ... markus this was one of your feedback, would you like to tackle it? 16:47:58 ... I guess manu had mentioned on a previous call that he wanted to tackle some amount of refactoring 16:48:07 markus_sabadello: I'm far less competant with this js architectural things than other people 16:48:14 Orie: let's assign manu, that'll be good. 16:48:58 ... proposal to create a suite for major sections of the spec... also assigning to manu 16:49:12 ... I know he doesn't wantto be critical path, but.. 16:50:15 ... we'll leave the producer tests should be of the ADM 16:50:49 ... Every issue is assigned a person 16:50:54 ... we have no open PRs 16:51:05 ... Anything we would like to discuss? 16:51:20 ... I have a suggestion if no-one else has anything 16:51:37 ... I'd like to talk about how we'll produce visualisation of the test results 16:51:42 https://github.com/w3c/did-test-suite/issues/51 16:51:47 ivan: seems a very elegant word... 16:51:55 Orie: could be a green/red checkmarks you've seen 16:51:58 ivan: exactly 16:52:01 ... don't overcomplicate things 16:52:02 Orie: okay 16:52:08 ... I've opened issue 51 and make a comment 16:52:18 ... minimum green checkmarks and red xs 16:52:35 ivan: what you have in such a document is the list of implementors where every test has the greens and reds 16:52:44 ... and the goal is that in eah row you would have at least two greens 16:53:04 ... now, having a human readable description or something you can pull out of metadata of what each tests does or what it is for is a good thing to have 16:53:09 ... and also a good thing to have a few words on a metadata level 16:53:15 ... for each implementation that you have 16:53:19 ... but that's it 16:53:24 Orie: awesome 16:53:32 ... wrote that 16:53:47 ... tried in the past to programatically... after we run the suites you get a json object which contains the test results 16:53:57 ... if you iterate that json object you can produce something close enough to a test report 16:54:02 ... it is frustrating to do that 16:54:07 ... it's a lot of html building from json objects 16:54:16 ... if the object payloads change stuff .. it can becme brittle 16:54:30 ... I more recently abandoned spending any time on this becuase we know the test shape is changing 16:54:48 ... I would hold off on trying any form of this until we feel really comfortable with the way the tests are feeling 16:54:56 ivan: yes, clearly a better way to do this 16:55:09 ... Did you have examples of other reports? 16:55:20 Orie: I looked at the VCs test suite report 16:55:40 ... we had been in other projects that use jest as the underlying test infrastrucutre there is some automated visualisation and reporting 16:55:49 ... but it is not super friendly from a reading perspective, but you get it for free 16:55:55 ... also an arguement that we should add that free reporting here 16:56:04 ... very little work to do it, another thing for folks to look at and it's automated 16:56:20 ivan: one example of one we use in a different wg 16:56:22 -> example for report https://w3c.github.io/publ-tests/test_reports/manifest_processing/index.html 16:56:44 ... is automated from json 16:56:47 ... not very exciting 16:57:00 ... not a big deal to do 16:57:56 Orie: this is very readable 16:58:06 https://github.com/w3c/publ-tests/tree/main/publication_manifest/manifest_processing/reports 16:58:20 ivan: one json file for each implementation 16:58:27 ... custom format 16:58:38 ... not complicated 16:58:45 https://github.com/w3c/publ-tests/blob/main/publication_manifest/manifest_processing/tests/index.json 16:58:53 ... and there is this json file which gives you data about each test 16:59:04 ... separate metadata for the test suite 16:59:08 ... and each implementer puts in a simple one 16:59:13 ... and javascript combines all that 16:59:33 Orie: we'll need something like this 16:59:40 ... we saw evidence of this in manu's PR around the did:key implementation 16:59:47 ... overloadng the describe block with metadata about the implementation 17:00:00 ... where perhaps we'd rather keep that describe block focussedon what the tests are about and pull that metadata from somewhere else 17:00:08 ... this looks good 17:00:12 ... Thank you all 17:00:28 rrsagent, draft minutes 17:00:28 I have made the request to generate https://www.w3.org/2021/04/01-did-topic-minutes.html ivan 17:00:38 thanks 17:11:08 Orie has gone... but ivan, another example of reports... https://linkedresearch.org/ldn/tests/summary which are all generated from / linked to the spec itself with copious amounts of RDFa, and all available as RDf using EARL :D 18:27:22 Zakim has left #did-topic