DID WG Topic Call on the Test Suite — Minutes
Date: 2021-04-13
See also the Agenda and the IRC Log
Attendees
Present: Manu Sporny, Brent Zundel, Charles Lehner, Daniel Buchner, Geun-Hyung Kim, Shigeya Suzuki, Juan Caballero
Regrets:
Guests:
Chair: Brent Zundel
Scribe(s): Charles Lehner, Brent Zundel
Content:
Brent Zundel: Getting started
… This is a meeting to discuss issues and problems with the test suite, as well as to work together to resolve them
Manu Sporny: I thought we could go over the two PRs, just one outstanding, need input from Orie and Charles.
… The other one, need confirmation from Markus
… Then may handle challenges in tests
Markus Sabadello: sounds good
Manu Sporny: Thoughts about discussion. Changes?
… Jumping in. Screen sharing.
Charles Lehner: https://github.com/w3c/did-test-suite/pull/54
Manu Sporny: 2020 did:key method
… Waiting for folks to approve
Markus Sabadello: Thought it was okay. But wondering if we want to merge implementation reports at this point or after. But either way is okay
Manu Sporny: You were saying we aren’t committing implementations yet?
Markus Sabadello: yes, formats may change … not finalized
Manu Sporny: Given I have been restructuring, am happy to keep these up to date with changes
… Will go ahead and merge
Charles Lehner: https://github.com/w3c/did-test-suite/pull/55
Manu Sporny: Addressing feedback
… Could change representation -> didDocumentStream
Orie Steele: want didDocumentStream. But it could be separate PR
Manu Sporny: will update this PR
… what do folks want for the key?
Markus Sabadello: representation more accurate
Orie Steele: should update DID Core then
Manu Sporny: We are about to get a bunch of implementations
Orie Steele: do what the spec uses. Why use a different term?
Manu Sporny: fine changing it, sorry markus_sabadello, like representation better but… any objections?
Markus Sabadello: section talking about didDocumentStream is only in resolution
… other sections talk about representation
Orie Steele: sad. why do we have didDocumentStream in spec at all?
… when I look at this, I expect it to be about resolution, therefore expect it to match
Manu Sporny: it is not, it is associated with the ADM
Orie Steele: could change spec …
Manu Sporny: that would be normative change
… would change implementations? or maybe wouldn’t.
Orie Steele: is the section on resolution normative?
Manu Sporny: yes, it’s left hand value
Orie Steele: I would expect document that is a DID document stream to have value didDocumentStream
Manu Sporny: but it’s not resolution
Orie Steele: is it a document encoded as a byte string?
Manu Sporny: section 6 - representations.
Orie Steele: I guess this is why better to write tests than specs
Charles Lehner: I don’t have strong opinions
… wasn’t aware of the discrepancy
Manu Sporny: happy to flip vote
Orie Steele: what is this section used for?
Manu Sporny: representation-specific entries, … data model core properties tests
Markus Sabadello: there is also did document metadata and resolution metadata in this fixture. is it needed there?
Manu Sporny: It might not be needed anymore. There are some tests using DID resolution metadata
Markus Sabadello: These tests should not need that
Manu Sporny: Some parts of the specification say must signal the media type in some way… there are two or three tests that use that
… Can change later … search and replace. Would anyone object to that?
Orie Steele: Would be fine with raising an issue
Manu Sporny: The other one… https://github.com/w3c/did-test-suite/pull/55#discussion_r611645272
… I considered that when writing the tests, but unsure if people want the same data model to apply to different ….
… representations or DID document streams or whatever you want to call it
Daniel Buchner: Don’t cross the representation streams
Manu Sporny: data model properties and representation-specific entries… Charles is proposing moving it up
… but I was unsure if people would want that. This allows us to have different data model representations per serialization (?)
… This mechanism is more flexible
Charles Lehner: can you clarify?
Manu Sporny: did+ld+json. representation-specific entries with context. in did+json, might have that but not expecting it to be preserved
… If we were to take this and move it up, then all of the sudden, our representation-specific entries would have other entries
… Raises questions. Whereas this… we have representation + representation-specific entries, and expect this result
… Concern, in the future will have other things
… representation-specific entries are meant to be for the representation being produced. I don’t think we have consensus on what should be in there, but spec should say what to preserve
Charles Lehner: ok, i think it’s fine
Orie Steele: Don’t know about others, but for JSON should preserve. But I would put them outside, and show in our implementation that we preserve …
… Each implementation can still show that they preserve whether inside or outside… but inside has more duplication
Markus Sabadello: Should be outside, because they are representation-specific entries
… It is called representation-specific entries so should be in the block. But the dmProperties should be outside
Manu Sporny: I want to point out that doing these kinds of changes isn’t actually changing the outcome of the test suite. Doesn’t need to be perfect, just need to make sure the statements are tested
…
Juan Caballero: I hesitate to have opinions on this because it is very complex. I think many people are on mute because it boils down to the translation issue that is confusing to a lot of people
Manu Sporny: Going to implement that. Heard Orie and Markus agree
… Then will merge the PR.
… markus_sabadello: do you want to talk about the tests that are challenging next?
Markus Sabadello: Yes.
Charles Lehner: https://github.com/w3c/did-test-suite/issues/53
Markus Sabadello: Something I haven’t implemented, because thought I would be able to use code that has now been merged… I will probably try to do another pass on the DID resolution where I try to reuse some of the other code
… I could test the resolve function, and what comes out of it could then run through production and consumption code to effectively test this statement in the resolution section
… That is my idea, why I didn’t initially implement some of these statements
… There are two or three of those. There is also one that says that one of the return values of the resolution function is DID document metadata, must be a valid metadata structure
… There are other tests now testing whether something is a valid metadata structure
… I will do another pass on the resolution tests to see if I can complete some of the missing ones here.
… There are also a few statements I don’t know how to test them.
… Related to equivalentId and canonicalId
… about what a DID method specification must do
… I don’t know how to write a test for it
Daniel Buchner: I think this goes down to a deeper point about so many parts of the spec. Of course we all accept, I hope, that DID methods have to do things that are not outwardly testable.
… How does it find the right keys? That is a method-specific question. Replete throughout the spec, there are countless such questions.
… Service endpoints tracking
… I don’t see this as any different
Manu Sporny: I agree with that, dbuc, I think it’s unfortunate that we put normative statements here for these two.
… But the thing that sells it for me is that it refers to DID method specifications.
… That is not machine-testable
Daniel Buchner: One thing that is testable, is you can write tests, obviously contrived…
… You can write a database in memory that updates a column with canonical value
… I can write a test that does the correct thing.
… One might use MongoDB, or MySQL. Queries will look slightly different
Manu Sporny: markus_sabadello: Are you okay with this?
… Yes, don’t need to write tests for these items.
Markus Sabadello: Maybe I will mention this.
Orie Steele: you can make the statement as .skip and add a comment in it pointing to the meeting minutes for the call.
Markus Sabadello: This one, Charles had a suggestion in the thread on how to test this, which I think will work
… This one, and this one, (pointing) … I will try to somehow test them using test code from other sections
… This one … must return in conformant representation … I think I know how to test it as well.
… That is the whole list; I will do another pass and report back on the issue
Manu Sporny: Charles, are there outstanding tests you had any challenges with?
Charles Lehner: I don’t think so
Manu Sporny: Orie
Orie Steele: I don’t think so
Daniel Buchner: I was, a little
Manu Sporny: Any of the other ones?
Orie Steele: I am in a holding pattern for my assigned issues
Manu Sporny: shigeya I think there was one? Were you able to address it or should we talk about it?
Shigeya Suzuki: I was observing the discussion on how to phrase the method-specific representation parameter thing earlier in this call.
… I am wondering how we can adapt for the did-spec-registry thing in my context
… Let me think about it and ask questions later. I think we can find a better way.
Manu Sporny: dbuc, you have the JSON tests assigned to you. Let me share …
Charles Lehner: https://github.com/w3c/did-test-suite/issues/26
Manu Sporny: JSON Production and Consumption … There are examples of what these tests look like
Daniel Buchner: I looked at the did-jsonld-production file …
… Inbound on that function there is an object with resolutionResult … don’t see same structure
Manu Sporny: It is old
… You can use JSON-LD production as a starting point
Daniel Buchner: Ok, I can do that
Manu Sporny: For consumption… I guess we haven’t had a case where we needed consumption tests.
Daniel Buchner: I also had a sort of general question about the JSON-only version in these tests. In the production I didn’t really see anything super-specific
… Other than it … has this media type
… Consumption, I wasn’t seeing about … bouncing values, even though justin_r on previous call said … need to bucketize …
Manu Sporny: Consumption is basically JSON.parse with making sure you are testing all these values as input. That’s really all consumption is.
Orie Steele: You’ll just struggle with numbers and dates, because there aren’t really any examples
… I don’t think anyone has shown a DID document that encoded a number, or null; it’s just not a thing
Manu Sporny: But there is … But the vast majority is JSON parse and stringify.
Daniel Buchner: The context value…. do I have to care? No but you have to do something to show you are doing something with it?
Manu Sporny: That’s sort of already there.
… The best thing to do is to take a shot at writing the tests.
… People will complain if they feel it is not right
… Let us know how we can help.
… That should be all of the tests, I believe. Every one we have been contemplating - modulo CBOR Production and Consumption which johnnycrunch has on his plate to do.
… Orie, the visualization thing, I think we are close
Orie Steele: Will kick it… until more to visualize
Manu Sporny: There are three
Orie Steele: did:example I don’t really count as real.
Manu Sporny: two did:key implementations
… Assigning it
… Other thing to bring up… dbuc since you are here, we ask people who are implementing equivalentId and canonicalId , didn’t get anyone except….
Daniel Buchner: Troy, did:orb,](did:orb,)` …
Manu Sporny: We just need two
… ion
Daniel Buchner: Don’t know… SecureKey…
Manu Sporny: We just need two independent implementations
… I think that’s it. Is there anything else test suite related
… that folks want to talk about?
… Reminder on roadmap and timeframing: we need tests done by end of this month
… Please do as soon as you can
… End of April all tests in, and of May all implementations in
Brent Zundel: That’s the goal, yes
Daniel Buchner: Are there more things on the agenda?
Manu Sporny: Not from me
Daniel Buchner: Can I ask one thing about the relative path stuff…
… It wasn’t immediately clear…. we were using a blank string because populated by JSON-LD and seemed fine…
… Is using just a pound sign okay?
… Could use fully qualified URLs. But … is ambiguous …
Orie Steele: You should switch to using fully-qualified URLs; everyone will thank you
… and you are probably already gzipping
… We messed up when we allowed relative refs in the DID document.
Daniel Buchner: Can switch it… We have people asking… It’s still in the spec. What is even allowed?
… On the web, pound sign if not populated doesn’t refer to.
Orie Steele: It’s the same, but then there are other normative statements
… Like about controller property, must be a DID
Orie Steele: https://w3c.github.io/did-core/#did-controller
Manu Sporny: To answer directly: nothing will change in the spec unless some raises a PR or issue
… We are locked in. If you feel additional language needed, can… But I suggest let’s not go there
Brent Zundel: Anything else?
… Thank you all for coming.
… Special call to be held weekly until test suites done.
… Let’s keep moving forward
Juan Caballero: thx all
Charles Lehner: You are welcome