DID WG Topic Call on the Test Suite — Minutes

Date: 2021-04-13

See also the Agenda and the IRC Log


Present: Manu Sporny, Brent Zundel, Charles Lehner, Daniel Buchner, Geun-Hyung Kim, Shigeya Suzuki, Juan Caballero



Chair: Brent Zundel

Scribe(s): Charles Lehner, Brent Zundel


Brent Zundel: Getting started
… This is a meeting to discuss issues and problems with the test suite, as well as to work together to resolve them

Manu Sporny: I thought we could go over the two PRs, just one outstanding, need input from Orie and Charles.
… The other one, need confirmation from Markus
… Then may handle challenges in tests

Markus Sabadello: sounds good

Manu Sporny: Thoughts about discussion. Changes?
… Jumping in. Screen sharing.

Charles Lehner: https://github.com/w3c/did-test-suite/pull/54

Manu Sporny: 2020 did:key method
… Waiting for folks to approve

Markus Sabadello: Thought it was okay. But wondering if we want to merge implementation reports at this point or after. But either way is okay

Manu Sporny: You were saying we aren’t committing implementations yet?

Markus Sabadello: yes, formats may change … not finalized

Manu Sporny: Given I have been restructuring, am happy to keep these up to date with changes
… Will go ahead and merge

Charles Lehner: https://github.com/w3c/did-test-suite/pull/55

Manu Sporny: Addressing feedback
… Could change representation -> didDocumentStream

Orie Steele: want didDocumentStream. But it could be separate PR

Manu Sporny: will update this PR
… what do folks want for the key?

Markus Sabadello: representation more accurate

Orie Steele: should update DID Core then

Manu Sporny: We are about to get a bunch of implementations

Orie Steele: do what the spec uses. Why use a different term?

Manu Sporny: fine changing it, sorry markus_sabadello, like representation better but… any objections?

Markus Sabadello: section talking about didDocumentStream is only in resolution
… other sections talk about representation

Orie Steele: sad. why do we have didDocumentStream in spec at all?
… when I look at this, I expect it to be about resolution, therefore expect it to match

Manu Sporny: it is not, it is associated with the ADM

Orie Steele: could change spec …

Manu Sporny: that would be normative change
… would change implementations? or maybe wouldn’t.

Orie Steele: is the section on resolution normative?

Manu Sporny: yes, it’s left hand value

Orie Steele: I would expect document that is a DID document stream to have value didDocumentStream

Manu Sporny: but it’s not resolution

Orie Steele: is it a document encoded as a byte string?

Manu Sporny: section 6 - representations.

Orie Steele: I guess this is why better to write tests than specs

Charles Lehner: I don’t have strong opinions
… wasn’t aware of the discrepancy

Manu Sporny: happy to flip vote

Orie Steele: what is this section used for?

Manu Sporny: representation-specific entries, … data model core properties tests

Markus Sabadello: there is also did document metadata and resolution metadata in this fixture. is it needed there?

Manu Sporny: It might not be needed anymore. There are some tests using DID resolution metadata

Markus Sabadello: These tests should not need that

Manu Sporny: Some parts of the specification say must signal the media type in some way… there are two or three tests that use that
… Can change later … search and replace. Would anyone object to that?

Orie Steele: Would be fine with raising an issue

Manu Sporny: The other one… https://github.com/w3c/did-test-suite/pull/55#discussion_r611645272
… I considered that when writing the tests, but unsure if people want the same data model to apply to different ….
… representations or DID document streams or whatever you want to call it

Daniel Buchner: Don’t cross the representation streams

Manu Sporny: data model properties and representation-specific entries… Charles is proposing moving it up
… but I was unsure if people would want that. This allows us to have different data model representations per serialization (?)
… This mechanism is more flexible

Charles Lehner: can you clarify?

Manu Sporny: did+ld+json. representation-specific entries with context. in did+json, might have that but not expecting it to be preserved
… If we were to take this and move it up, then all of the sudden, our representation-specific entries would have other entries
… Raises questions. Whereas this… we have representation + representation-specific entries, and expect this result
… Concern, in the future will have other things
… representation-specific entries are meant to be for the representation being produced. I don’t think we have consensus on what should be in there, but spec should say what to preserve

Charles Lehner: ok, i think it’s fine

Orie Steele: Don’t know about others, but for JSON should preserve. But I would put them outside, and show in our implementation that we preserve …
… Each implementation can still show that they preserve whether inside or outside… but inside has more duplication

Markus Sabadello: Should be outside, because they are representation-specific entries
… It is called representation-specific entries so should be in the block. But the dmProperties should be outside

Manu Sporny: I want to point out that doing these kinds of changes isn’t actually changing the outcome of the test suite. Doesn’t need to be perfect, just need to make sure the statements are tested
. Is this correct?

Juan Caballero: I hesitate to have opinions on this because it is very complex. I think many people are on mute because it boils down to the translation issue that is confusing to a lot of people

Manu Sporny: Going to implement that. Heard Orie and Markus agree
… Then will merge the PR.
… markus_sabadello: do you want to talk about the tests that are challenging next?

Markus Sabadello: Yes. … I implemented tests for DID resolution and dereferencing. There are some statements I am not testing yet. … Summarizing in an open issue. Some reference other parts of the specification … Such as production and consumption that we just talked about

Charles Lehner: https://github.com/w3c/did-test-suite/issues/53

Markus Sabadello: Something I haven’t implemented, because thought I would be able to use code that has now been merged… I will probably try to do another pass on the DID resolution where I try to reuse some of the other code
… I could test the resolve function, and what comes out of it could then run through production and consumption code to effectively test this statement in the resolution section
… That is my idea, why I didn’t initially implement some of these statements
… There are two or three of those. There is also one that says that one of the return values of the resolution function is DID document metadata, must be a valid metadata structure
… There are other tests now testing whether something is a valid metadata structure
… I will do another pass on the resolution tests to see if I can complete some of the missing ones here.
… There are also a few statements I don’t know how to test them.
… Related to equivalentId and canonicalId
… about what a DID method specification must do
… I don’t know how to write a test for it

Daniel Buchner: I think this goes down to a deeper point about so many parts of the spec. Of course we all accept, I hope, that DID methods have to do things that are not outwardly testable.
… How does it find the right keys? That is a method-specific question. Replete throughout the spec, there are countless such questions.
… Service endpoints tracking
… I don’t see this as any different

Manu Sporny: I agree with that, dbuc, I think it’s unfortunate that we put normative statements here for these two.
… But the thing that sells it for me is that it refers to DID method specifications.
… That is not machine-testable

Daniel Buchner: One thing that is testable, is you can write tests, obviously contrived…
… You can write a database in memory that updates a column with canonical value
… I can write a test that does the correct thing.
… One might use MongoDB, or MySQL. Queries will look slightly different

Manu Sporny: markus_sabadello: Are you okay with this?
… Yes, don’t need to write tests for these items.

Markus Sabadello: Maybe I will mention this.

Orie Steele: you can make the statement as .skip and add a comment in it pointing to the meeting minutes for the call.

Markus Sabadello: This one, Charles had a suggestion in the thread on how to test this, which I think will work
… This one, and this one, (pointing) … I will try to somehow test them using test code from other sections
… This one … must return in conformant representation … I think I know how to test it as well.
… That is the whole list; I will do another pass and report back on the issue

Manu Sporny: Charles, are there outstanding tests you had any challenges with?

Charles Lehner: I don’t think so

Manu Sporny: Orie

Orie Steele: I don’t think so

Daniel Buchner: I was, a little

Manu Sporny: Any of the other ones?

Orie Steele: I am in a holding pattern for my assigned issues

Manu Sporny: shigeya I think there was one? Were you able to address it or should we talk about it?

Shigeya Suzuki: I was observing the discussion on how to phrase the method-specific representation parameter thing earlier in this call.
… I am wondering how we can adapt for the did-spec-registry thing in my context
… Let me think about it and ask questions later. I think we can find a better way.

Manu Sporny: dbuc, you have the JSON tests assigned to you. Let me share …

Charles Lehner: https://github.com/w3c/did-test-suite/issues/26

Manu Sporny: JSON Production and Consumption … There are examples of what these tests look like

Daniel Buchner: I looked at the did-jsonld-production file …
… Inbound on that function there is an object with resolutionResult … don’t see same structure

Manu Sporny: It is old
… You can use JSON-LD production as a starting point

Daniel Buchner: Ok, I can do that

Manu Sporny: For consumption… I guess we haven’t had a case where we needed consumption tests.

Daniel Buchner: I also had a sort of general question about the JSON-only version in these tests. In the production I didn’t really see anything super-specific
… Other than it … has this media type
… Consumption, I wasn’t seeing about … bouncing values, even though justin_r on previous call said … need to bucketize …

Manu Sporny: Consumption is basically JSON.parse with making sure you are testing all these values as input. That’s really all consumption is.

Orie Steele: You’ll just struggle with numbers and dates, because there aren’t really any examples
… I don’t think anyone has shown a DID document that encoded a number, or null; it’s just not a thing

Manu Sporny: But there is … But the vast majority is JSON parse and stringify.

Daniel Buchner: The context value…. do I have to care? No but you have to do something to show you are doing something with it?

Manu Sporny: That’s sort of already there.
… The best thing to do is to take a shot at writing the tests.
… People will complain if they feel it is not right
… Let us know how we can help.
… That should be all of the tests, I believe. Every one we have been contemplating - modulo CBOR Production and Consumption which johnnycrunch has on his plate to do.
… Orie, the visualization thing, I think we are close

Orie Steele: Will kick it… until more to visualize

Manu Sporny: There are three

Orie Steele: did:example I don’t really count as real.

Manu Sporny: two did:key implementations
… Assigning it
… Other thing to bring up… dbuc since you are here, we ask people who are implementing equivalentId and canonicalId , didn’t get anyone except….

Daniel Buchner: Troy, did:orb,](did:orb,)` …

Manu Sporny: We just need two
… ion

Daniel Buchner: Don’t know… SecureKey…

Manu Sporny: We just need two independent implementations
… I think that’s it. Is there anything else test suite related
… that folks want to talk about?
… Reminder on roadmap and timeframing: we need tests done by end of this month
… Please do as soon as you can
… End of April all tests in, and of May all implementations in

Brent Zundel: That’s the goal, yes

Daniel Buchner: Are there more things on the agenda?

Manu Sporny: Not from me

Daniel Buchner: Can I ask one thing about the relative path stuff…
… It wasn’t immediately clear…. we were using a blank string because populated by JSON-LD and seemed fine…
… Is using just a pound sign okay?
… Could use fully qualified URLs. But … is ambiguous …

Orie Steele: You should switch to using fully-qualified URLs; everyone will thank you
… and you are probably already gzipping
… We messed up when we allowed relative refs in the DID document.

Daniel Buchner: Can switch it… We have people asking… It’s still in the spec. What is even allowed?
… On the web, pound sign if not populated doesn’t refer to.

Orie Steele: It’s the same, but then there are other normative statements
… Like about controller property, must be a DID

Orie Steele: https://w3c.github.io/did-core/#did-controller

Manu Sporny: To answer directly: nothing will change in the spec unless some raises a PR or issue
… We are locked in. If you feel additional language needed, can… But I suggest let’s not go there

Brent Zundel: Anything else?
… Thank you all for coming.
… Special call to be held weekly until test suites done.
… Let’s keep moving forward

Juan Caballero: thx all

Charles Lehner: You are welcome