DID WG Topic Call on the Test Suite — Minutes

Date: 2021-04-29

See also the Agenda and the IRC Log

Attendees

Present: Manu Sporny, Brent Zundel, Orie Steele, Shigeya Suzuki, Markus Sabadello, Ted Thibodeau Jr.

Regrets:

Guests:

Chair: Brent Zundel

Scribe(s): Brent Zundel, Manu Sporny

Content:


Brent Zundel: Let’s dive in…

1. DID test suite

Manu Sporny: https://github.com/w3c/did-test-suite/pulls

Manu Sporny: https://github.com/w3c/did-test-suite/pull/70

Manu Sporny: PRs to review, Shigeya’s looks good. I think we should pull it in.
… I still don’t know what the docker image does.

Orie Steele: for folks who want to run the test harness as a web server, then you can send JSON to it.
… if someone wanted to host it, that would allow real-time DID testing

Manu Sporny: is there an example of how to do it?

Orie Steele: there was. I expect it should still work.

Manu Sporny: I don’t expect it to work. I am skeptical that it will work after the restructuring. Is there anything in the readme?

Orie Steele: there was, at one point we hosted a version of it. possible the restructuring broke it.

Manu Sporny: shigeya have you been able to test that this works?

Shigeya Suzuki: it worked as expected
… I agree with Orie

Manu Sporny: https://github.com/w3c/did-test-suite/pull/71

Manu Sporny: other PR is initial JSON test, is buchner here?
… hopefully he got his answers. Can’t merge until tests are done.

Manu Sporny: https://github.com/w3c/did-test-suite/pull/71/files#diff-a311c8ed2431c0662a73a2e224d5fed7edfd946d8c201df6bc7f1fb877b609a9R31-R35

Manu Sporny: we don’t have any boolean or null properties, not sure if there are sequence values where order is important.
… it may be that we have some tests for which there aren’t test values

Orie Steele: https://did.actor/mike/

Orie Steele: https://did.actor/mike/did.json

Orie Steele: ^ a did document that uses features nobody else uses.

Manu Sporny: it is possible that two implementations could submit a test with unregistered properties that have these value types to test them
… would there be objections to that?
… Orie has one, we would need one more
… I don’t think this really needs to be tested, even if we leave the normative statements in, but the easiest thing is to get two examples in.

Orie Steele: these features need to be tested or removed.

Manu Sporny: if Ories submits did mike, then daniel buchner can submit something as well. We can point him to this conversation and ask him to submit an implementation that tests them.
… Amy has created a checklist for all normative statements

Manu Sporny: https://github.com/w3c/did-test-suite/issues/72

Manu Sporny: https://github.com/w3c/did-test-suite/issues/73

Manu Sporny: https://github.com/w3c/did-test-suite/issues/74

Manu Sporny: https://github.com/w3c/did-test-suite/issues/75

Manu Sporny: https://github.com/w3c/did-test-suite/issues/76

Manu Sporny: https://github.com/w3c/did-test-suite/issues/77

Manu Sporny: wee need to assign these to people, this will let us be sure all the normative statements are tested.
… who did syntax and parameters?

Markus Sabadello: I did some

Manu Sporny: I will take data model
… production and consumption are all together, so this will be daniel (JSON), me (JSON-LD), and Jonathan (CBOR)
… markus is resolution

Markus Sabadello: yes

Manu Sporny: did methods are all human testable, I’ll assign Drummond
… once we go through the checklists we will know if we have all the tests we need.
… need to be done before our next tuesday call
… any concerns or objections?

Brent Zundel: not hearing any

Manu Sporny: https://github.com/w3c/did-test-suite/issues/51

Manu Sporny: last item, @issue51
… we have 5? implementations

Orie Steele: There’s not enough data to justify investing in visualization

Manu Sporny: I disagree, we have 9 implementations, when is it enough?

Orie Steele: if we have 9 actual separate ones, that would be enough. Last I checked it wasn’t enough.

Manu Sporny: there are 5 methods and 3 resolvers and 1 dereferencer

Orie Steele: at a minimum we need two of each

Manu Sporny: but we still need a report
… we need at least one of each thing. I suggest we have enough. If you disagree, I want to know how much we need.

Orie Steele: there was automated reporting set up. It should be possible, but I am hesitant to do it with only 1 of a thing.

Manu Sporny: the only thing that could change at this point is adding or removing tests, I hope the test generator doesn’t care.
… I don’t think it’s wasted effort to do a report.

Orie Steele: generating the report it trivial, anyone can do it.
… the question is if we should do it now

Manu Sporny: I would prefer that

Orie Steele: at what point is the report visualized enough. we have the automated one, I did an implementation that was better, but it is possibly broken since the refactor

Manu Sporny: that’s the one I mean, I’m fairly sure its broken, could you fix it?

Orie Steele: if tests change, that report breaks

Manu Sporny: that means we’ll have to wait. I was hoping we could get it to a stage where we can see how far off we are.
… without having to manually do it.
… that’s not ideal

Orie Steele: the code is there, anyone can update it.

Manu Sporny: it was hard for me to understand it. Not clear to me how the tallying was happening
… looking at the package.json

Orie Steele: follow the link at the top level readme will lead you to throw to generate the test report.
… it would do well to have another developer look at it

Manu Sporny: then we will need someone to step forward.

Orie Steele: I at least need someone else to work with

Manu Sporny: we’ll put that out to the group. We need that for visibility in how we’re doing
… any other issue to discuss?

Orie Steele: https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server

Markus Sabadello: which part of the code is about the reports?

Orie Steele: https://github.com/w3c/did-test-suite/blob/main/packages/did-core-test-server/package.json#L11

Manu Sporny: https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server#usage

Orie Steele: https://github.com/w3c/did-test-suite/blob/main/packages/did-core-test-server/generate-latest-respec-data.js

Markus Sabadello: thanks

Orie Steele: https://github.com/w3c/did-test-suite/blob/main/index.html#L686

Orie Steele: ^ rally this is the problem

Orie Steele: report is generated client side from json in respec thats highly brittle

Brent Zundel: I will send an email to the mailing list that points folks to github

Brent Zundel: action brent will send an email to the list