DID WG Topic Call on testing — Minutes
Date: 2021-03-5
See also the Agenda and the IRC Log
Attendees
Present: Shigeya Suzuki, Markus Sabadello, Manu Sporny, Brent Zundel, Charles Lehner, Justin Richer, Orie Steele, Amy Guy
Regrets:
Guests:
Chair: Brent Zundel
Scribe(s): Manu Sporny, Brent Zundel
Content:
Brent Zundel: This might be a quick call.
Markus Sabadello: We could cover some resolution and dereferencing tests if we have time to do that.
Brent Zundel: I think that sounds like a good idea.
Manu Sporny: +1
Brent Zundel: Let’s jump in.
1. Resolution and Dereferencing Tests
Markus Sabadello: Let’s me try to see if I understand the test suite correctly.
… I’ve been looking at the PRs
… There are these default.json and defaultSuiteConfig.json that are inputs and the tests are run against those.
… I don’t understand why we have both files.
Manu Sporny: One of them is for human readable tests, other is for npm run test.
Markus Sabadello: One of them is in an array, the other one is not.
… My understanding is this is the input data being tested, then there are JS files with test files.
… I started to work on separate suites for did resolution and url dererefencing.
… I’m not sure if that’s the right thing to do — I put did-resolution and did-url-dereferencing into their own suites… all works in progress.
… I haven’t started working on the test code yet, started to work a bit on what input data should be… for resolution and would like to share what I’m thinking.
… Want to see if it makes sense for resolution — I’m thinking there is an array called executions… resolve in did spec … array, for each execution, there is input and output
… There is an array for resolve and resolveRepresentations… one main difference, executing resolve returns a did document data model, whereas resolveRepresentation returns a DID Document stream… in one of the concrete representations.
… With this input data, I can write code for all normative statements… I could test that input DID must match the ID property in the resulting DID Document and other things.
… So, accept, content type, etc.
… One additional thing — trying to annotate and mark executions as returning as being expected to return certain errors.
… I’m saying that the second execution in this array, which would be notFoundExecutions, input that is not found, therefore output has an error… I can see if test code sees if this is true.
… I could write test that checks, is the value not found, does it contain an error, etc.
Manu Sporny: All of this sounds good, structure seems sane, you’re testing in a way that makes sense… I’d just worry that Orie had a different idea of how this might be done.
Markus Sabadello: My concern is that suites in VC HTTP API is groups of functionality… might make sense to do that here as well.
… The tests seem to operate on these defaultSuiteConfig, would have expected tests to operate on fixture files… maybe that’s where input data will go.
… Maybe Orie was trying to reorganize this somehow… anyway, at an early stage trying to figure out input data and output data.
… There will be some relation with other tests… Manu, you had a comment — some testing functionality might be nested or reference each other. For example, DID Resolution says output of resolve must be possible to run that through one of the production algorithms to get to a conformant algorithm.
… The abstract DID Document data model, might need to take that and somehow model a production step and then test if the result is a conformant did document in conformant representation.
Charles Lehner: This looks good — wonder if it’s redundant when errors are already in the output.
Markus Sabadello: The idea is that will be tested… that says what executions have what sort of errors, and the result checks to see if that’s true.
… That says if the error happened.
Manu Sporny: the conversion stuff, two things: the consumption tests are really difficult, you will actually need a consumer.
… perhaps two people can write those tests, there aren’t too many
… we almost need to make pseudo implementations, not too concerned about that.
… just highlighting that. For example you have a DID Doc data model and call JSON stringify on it
… that would be one way to not re-use code elsewhere.
… we don’t want the test to be part of a larger series. You could throw JSON Schema at it, but then you test a lot of things in addition to the normative statement
… the other thing is that it has been fairly easy to write JSON and JSON-LD tests. We haven’t written any CBOR
… the current suite would require you to encode CBOR in JSON. The CBOR stuff may be a bit challenging.
Orie Steele: Here are our tests for jsonld consumption: https://github.com/transmute-industries/did-core/blob/main/packages/did-ld-json/src/did-ld-json.test.ts#L21
Orie Steele: We had to implement part of the ADM to test consumption
Markus Sabadello: regarding the encoding… trying to come up with resolve representation function data format — thought for plain JSON content type — could just do a string.
… JSON and JSON-LD are easy — resolve representation returns JSON-LD… just stringified JSON… for cbor, I used hex encoding.
… but I haven’t written any tests for that — quickly summarize for Orie
Orie Steele: Imo the hex encoding for cbor looks correct to me
Markus Sabadello: Had some questions for Orie… started to think about Resolution and Dereferencing — one question, the way I did it was by creating a different suite — not to put it into main DID spec.
… How does default.json and defaultSuiteConfig.json work?
Orie Steele: code splitting is good! easier to test independently
Orie Steele: I think Markus, you’ve nailed the how to test encoded variants… string encoded JSON, string encoded JSON-LD, hex-encoded CBOR, that’s great.
… Tests that iterate over fixture look at content type, decode content type, then decode properties.
… Your question regarding default suite config vs. default.json — the test suite is still not super well organized in that regard… I think we’re using JSON config files here where we should be using JS file that imports smaller files.
… default.json and defaultSuiteConfig.json are only different in external web server… one is an array and one is an object… if you’re asking hosted version of test suite to run on suite config — you pass suites you want to run, vs. just run individual suite — but I think that’s confusing.
… Most people are not running these tests in hosted mode — each org could host suite on test server.
… We should get rid of one of them
… There is a feature here where you can host the test suite on a web server and that’s what these are a test of.
Manu Sporny: +1 to removing defaultSuiteConfig.json
Markus Sabadello: Another question I had — different suite, different input file format… default suite config for existing tests — is not enough for me to test resolution and dereferencing.
… it doesn’t have input/ouput data, doesn’t have serialized items, but I think I might have created some redundancy — media types, mime types,
… maybe this is temporary
Orie Steele: I think you did the right thing — you want tests to be narrowly focused… test everything in one file, all possible options.
… First version of this test suite, tried to test everything w/ one config in one file… hard to maintain, not easy to extend.
… I think your approach works, is the right direction.
Markus Sabadello: One part of it that’s challenging, if you’re a DID Method implementer, then you’re going to have to go around to a bunch of different files to configure everything correctly… or maybe they don’t support DID Resolution or DID URL Dereferencing.
… Ok, then thanks very much, I haven’t looked at many tests yet… feel confident that this can be done, will continue with that over next couple of day.s
Manu Sporny: a conforming producer and consumer language is har dto test.
… one way is to focus on JSON-LD and assume that if there is an @context, they had to have it in their implementation
… nearly impossible without code analysis of running code. This is pseudo=testable but got through. all I did was take a best effort at writing the test, then wait for others to disagree
… maybe we can figure a way to test it more accurately
Markus Sabadello: I would probably say it’s not quite good enough, doesn’t test what we’re saying — representation-specific entries, my assumption was test would model these different maps, would also need to be in the fixtures… map of properties, representation specific entries, as a result, here is JSON-LD specific… test checks given these two input maps, is result correct.
Manu Sporny: we may end up discovering these things along the way. That is only testable in JSON-LD
… that’s probably okay, all we need are two independent implementations
… I’ll take your input and modify these tests
… it is more accurate to do it that way
… I’m a bit concerned about how many of these things happen. Perhaps we will need (at some point ) to refactor the test suite
Orie Steele: regarding refactoring the test suite, it’ll happen — many times
… We’ll need to increase convenience for Editors, test writers, etc… it’ll get refactored along the lines of what PRs get into it. We’re not saying any of the structure is set in stone.
… As test report starts to cover more and more of the spec, people will become increasingly cranky of refactoring the test suite…
… Rapid iteration on tests is critical, I expect we will change the way suite will work, inputs, etc.
… Folks should be less concerned about “will this be ok forever” vs. “are we getting contributions at a pace that will yield results for us.
Markus Sabadello: +1 to Orie
Brent Zundel: +1
Manu Sporny: +1 to what Orie just said.
Shigeya Suzuki: +1 now understand the mode.
Manu Sporny: could we spend a bit of time on understanding predicting what the long pole in the tent is going to be?
… I’m looking at the tests, Orie, what are your plans for merging the spec tests in?
Orie Steele: not planning on it
Manu Sporny: Markus is progressing, I’m nearly done, Charles?
Charles Lehner: I need to check in about that. I have a question
… There is some phrasing about ADM
… That a process needs to return that… wonder if it’s going to be ok to, how should it be represented in JSON config? Can we just use JSON representation, or special way to encode ADM?
Orie Steele: Our fixtures are in JSON and the good news is that JSON can represent the ADM — the answer is “yes”, if you need to produce an ADM, you need to represent it as JSON and be careful about how you communicate that…
… Just make sure you’re clear with name of variable… state is as didDocumentAdm or something like that.
… So everyone is super clear as JSON — we have to represent it as something.
Markus Sabadello: The way I’ve done it so far is, when I’m dealing in ADM, I’ve been using JSON — ADM can be expressed as JSON, when I’m trying to express concrete representation, I’m doing it as concrete reprenstation.
… (of that)
… But sometimes, tests have to test JSON and JSON-LD representation, in test suite… we use JSON to express ADM sometimes, and sometimes we express it as JSON-LD (as JSON) — we just want to be explicit w/ naming of variables.
Orie Steele: yes, what a markus is saying is also good…. i think its good to use strings for concrete representations as much as possible
Orie Steele: just be clear about what you are representing.
Charles Lehner: Ok, great.
Orie Steele: we get to drop features that are not tested right?
Manu Sporny: I’m a bit concerned about Daniel Buchner. They aren’t difficult, but they need to get done.
… sounds like we may have these in in a week or so
… the only other thing is folks getting their DID methods in as well
… Digital Bazaar plans to do a couple
… it may be a good idea to have the test suite re-factored before then
… order of operation is get suites in, then refactor, then get implementations in
… I think we can be done by the end of April
… in theory two more weeks
… thoughts?
Orie Steele: 1. get close to 100% converage….2. refactor test suite….. 3. add did methods
Orie Steele: Yep, generally +1 — we should get coverage reporting as close to 100% as we can before trying to do a big refactor.
… Make a new suite that’s very focus, like Markus did.
… DOn’t expect me to be the one that does the test suite refactoring.
… Then we can add DID Methods after that.
Manu Sporny: all that sounds good. There is one part I would like to refactor that will help folks submit their DID Methods
… not really more than that
… anything else people want to see done?
Orie Steele: Inevitably, people won’t write tests for features.
Manu Sporny: one caveat, if Dan Buchner doesn’t write the JSON I will have to
… I don’t know if Jonathan is going to write CBOR
… I thiink we have full coverage
Charles Lehner: relative DID URLs?
Orie Steele: should be able to look at a report
Manu Sporny: not sure how to do that. we’ll need to go line by line through the spec
… for example, MArkus caugfht that I hadn’t updated the normative sections for production
… people are contributing
… are you expecting something will be dropped?
Orie Steele: here is an example of such a report: https://transmute-industries.github.io/did-core/
Orie Steele: we should have something like this
Orie Steele: or declare failure.
Orie Steele: there needs to be a report so we know what’s not tested.
Markus Sabadello: I’m not guaranteeing I can write normative tests for everything… lots of statements — nextVersionId, last Update, I’ll do my best, but we’ll see
Orie Steele: the test suite outputs JSON, it shows you if its tested… or not.
Manu Sporny: there are a class of things that may be dropped
Orie Steele: the report better be automatic
Orie Steele: :)
Manu Sporny: canonicalID for example.
… sup[er easy to cheat the test suite
… I’ve seen it happen multiple times
… folks can’t point to an implementation
… not sure how exacting the group will be on that
… not sure how many people are learning to implement equivalentID
Orie Steele: We need to clearly communicate what is and is not tested… regarding statements and features.
Manu Sporny: that’s the class of things that will end up being removed. Not because tests aren’t written, but because implementations aren’t there
… I don’t know how to regenerate the output
Orie Steele: should be in the readme
… similar to other test suites
… all automatic
Manu Sporny: I couldn’t get it working
… I will read the readme
Brent Zundel: Anything else for today?
Orie Steele: https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server#usage
Orie Steele: if the readme is not correct, PRs welcome to fix it :)
Brent Zundel: Ok, that’s a meeting, thanks all for coming - thanks everyone for your work on the test suite, you’re all fantastic.