Meeting minutes
DID Resolution test infrastructure
wip: Today we continue the previous discussion about DID Resolution test infrastructure
wip: Goal is that we have a clear sense what test statements we want to create
manu: If we could get to a single test, what is the minimum viable test, that would be good.
manu: "Resolve a DID" is the minimal viable test, you should be able to make a call to the API and get a DID document back
manu: We will also want a negative test for that, a DID that fails to resolve
manu: That's the baseline
manu: I think we agreed that we wanted the test suite to be DID method agnostic, i.e. the DID resolver configuration will explictly tell you which DIDs to use for success, and which DIDs to use for failure
manu: The DID Resolver configuration will include the URL of the resolver, and DIDs for valid and invalid resolutions
manu: I think this is the minimal viable configuration
peacekeeper: I agree with all of that.
peacekeeper: We can set up an initial test, one or two of them, it'll be easy to contribute additional test cases after that. There are different reasons why resolution can fail, we have all these parameters/options/metadata that we can test, but that can all be easy to add once we get the basic infrastructure built.
Wip: This all sounds good, minimal viable test case a simple DID. People who submit the configuration would also submit the test DIDs.
Wip: For the negative test case, we should pick a "fail" case for minimal viable test suite
manu: Test #1 is successful resolution. Test #2 is unsuccessful resolution.
manu: You said the resolution result would be provided. I think that wouldn't be done, because this way it could be "cheating".
Wip: The test will still execute the resolution and check the result.
peacekeeper: I think people would not submit the entire resolution result; people would submit one or more DIDs, with expected failure -- not found, invalid, and people would submit resolution result, test suite will then test against that.
<JoeAndrieu> +1 for providing both input and output for each test
peacekeeper: There is a DID that would be known to not exist, then resolution result, and test suite will test if resolution result will be for DID that doesn't exist.
Wip: In some cases, maybe we don't people to provide the resolution result, because that should be standard.
Wip: If you execute any resolver on a DID that is not found, the resolution result should be the same.
Wip: If you execute resolution on a DID that is expected to resolve, then the resolver should return the expected result.
Wip: I provide a DID that is expected to resolve, the test executes the resolve function and get a resolution result.
manu: We should not be testing the actual DID method. We should only test DID resolution.
manu: That should happen as part of a DID method conformance test suite.
manu: We don't want this to become complicated.
manu: If we have people provide what the expected result is, this can lead to cheating.
manu: There are cases where people provide non-conformant DID document, and then we get into arguments about what is conformant.
manu: We don't want test suite maintainers to be in this position.
manu: We should draw the line a little closer to DID Resolution than what you are suggesting Will.
manu: First reason is we don't want test suite config files to become too complicated. So we don't want people to provide entire resolution responses.
manu: Second reason is we don't want to police the resolution results.
manu: All we want to test is things like HTTP error codes, if the resolution result is well-formed, etc.
manu: We can test the structure of what is returned, but not the deep content.
<Zakim> JoeAndrieu, you wanted to mention formally defining equivalence for result values
JoeAndrieu: I'm more aligned with Wip. We're testing the resolver, so we need to be checking the output.
JoeAndrieu: If we don't test that we're not checking if the resolver is actually resolving something.
JoeAndrieu: Regardless of whether we are going to test more deeply the values of the content (I think we should), we should at least lint or schema-verify the returned DID documents.
JoeAndrieu: Checking the headers is not enough.
Wip: manu convinved me that I don't think people have to provide the bare expected resolution result.
Wip: If the DID resolves successfully, we test if the resolution result contains a conformant DID document.
Wip: A resolver must produce a valid DID document response.
<Zakim> manu, you wanted to ask "define output" and to note "generic introspection"
manu: +1 to that. I'm fine with generic tests, e.g. is it a conformant DID document.
manu: E.g. if you resolve a DID, the returned DID document must contain an "id" with that DID. This is a generalized test.
manu: What I don't want to see is people telling us what a valid response looks like.
manu: The person getting tested should not tell us what the correct answer is.
manu: Generic tests are fine.
<Wip> +1 that makes sense to me
manu: We should not do DID-method specific tests.
manu: I expect this can be done with generalized JSON schemas.
manu: Or with generalized code.
Wip: With this, we should be able to build generalized infrastructure.
Wip: We're only going to test generalized things that should work across any DID method.
Wip: This should allow us to achieve interoperability.
<JoeAndrieu> +1 to support that. it's a strong interoperability argument
Wip: Let's discuss infrastructure.
manu: +1, yes, agreed, strong interop argument
Wip: What would be the process for submitting a resolver.
manu: I volunteered Digital Bazaar to re-use test test infrastructure we already have.
manu: First implementation could be Markus' resolver.
manu: We would be able to put one together e.g. for did:key
manu: I don't know how much of the DID Resolution spec we'd implement. There are a number of features we may not implement right now.
manu: We might want to chat with others who are doing resolvers.
manu: This should be enough to get us going.
manu: We can set up regular Github action cadences.
manu: Markus can provide an initial resolver implementation.
Wip: I don't think it should be hard to find people who submit test cases
Wip: Anyone working on a DID method should be able to provide a resolver.
Wip: DCD is working on did:btc1, may provide a Docker wrapper or endpoint
Wip: I would like to be part of the discussion when DB sets up the infrastructure.
bigbluehat: I wanted to point out there is already a did:key test suite. It's not been touched in about two years, but we may be able to reuse something.
manu: The did:key test suite is probably so old that it has patterns we don't want to follow. We probably want to do something that is more modern test infrastructure.
manu: That is a DID method test suite, the DID resolution test suite would have a somewhat different structure.
manu: It might just take as much time to adjust that existing test suite, as starting a new modern one.
manu: Happy to do knowledge transfer with Wip to do knowledge transfer, with ongoing cadence.
manu: We will have to get very specific with regard to who will be doing the work.
manu: If we want some knowledge transfer, let's get it down in minutes like this so we can point back to it.
manu: Or we can do CCG infrastructure to do recordings and minutes.
manu: Might be best to have special topic calls every other week, including engineers who will do the work on the minimal viable test suite.
bigbluehat: One question I had was whether we will have a did:key method test suite at some point. I can't speak to the quality, but it's using the same plumbing and reporting and Github actions.
bigbluehat: We could either fork or create a new repo. My expectation is we can strip it down to a single initial DID Resolution test suite. The plumbing should be identical to other test suites.
bigbluehat: If we do a did:key one at some point, we will build on the existing one.
Wip: We should use this time for calls.
Wip: I was thinking we do it un-minuted. But we could also do it as a CCG call that would be recorded.
Wip: Let's push hard to get the first version up and running.
manu: +1 to every other week, feels like a good cadence.
manu: We should task one of our engineers to get this up and running.
manu: Should we do it in CCG? As a CCG work item, we have video recording, screenshare, point to code, etc. Those things are vitally important for knowledge transfer.
manu: That's what's nudging me towards CCG. The downside there may be the IPR protection.
manu: But the test suites are already under W3C, so I'm not that concerned.
manu: I'm leaning pretty strongly to CCG minuting.
manu: We had our first Google Meet failure in CCG last time, we lost the recording and it's unrecoverable. Looks like it didn't even attempt a recording.
manu: Looks like it worked yesterday. Failures can happen, and you don't know until after the call is over.
manu: We should still do it in CCG.
Wip: I'm also leaning towards CCG.
Wip: Should be all open-source already.
Wip: Could it be done as CCG work item and then bring it into DID WG?
manu: +1. I'd suggest no making a CCG work item and bouncing it back and forth. We can just set up a CCG call where people can join and contribute.
manu: Test suites are not normative, and IPR is clean.
manu: I don't think we need to ask for permission or anything. We should notify Pierre-Antoine of this approach.
manu: We're just leveraging CCG infrastructure to use video recordings.
manu: Test suites are an official working group thing.
manu: If all people doing the work are in the DID WG, we shouldn't have to do anything.
Wip: I had this question about who can participate. I like your approach.
<Zakim> JoeAndrieu, you wanted to suggest a CCG educational webinar on how to contribute
Wip: People will want to participate in order to get their resolver into the e
Wip: People will want to participate in order to get their resolver into the test suite
JoeAndrieu: From an IPR perspective, if we do it under CCG, we should be covered.
JoeAndrieu: WGs have different IPR entaglement.
JoeAndrieu: Everything contributed under CCG should be covered appropriately.
JoeAndrieu: We should have a dedicated CCG call about how to contribute.
<TallTed> "joint meeting of CG & WG", as FedID does regularly
manu: I appreciate where JoeAndrieu is coming from. I don't want to add more work to our plates.
manu: I think maybe JoeAndrieu that might be a separate thing we do in CCG. The audience is different.
manu: People who are putting the test suite together must communicate directly.
manu: We want to avoid questions to derail the actual process. Those who are building the test suite should be enabled and protected.
Wip: I like where JoeAndrieu is coming from too. Maybe first we as a WG do a WG-focused engineering call. After that, we do more open sessions where we invite people to add resolvers and contribute.
Wip: I'm sure people want their DID method to be included.
<JoeAndrieu> +1 to getting the basics in place before a CCG call
Wip: Let's say we do this every 2 weeks, how long might this take us?
bigbluehat: Replicating any of the foundational test suites will get you up and running, including JSON and HTML. It's just about stripping down the repo to those minimal tests we want to run.
bigbluehat: And have one or two implementations we can test against.
bigbluehat: Once that's in place, we will have HTML and JSON output generated.
bigbluehat: Then it's just updates.
bigbluehat: The hardest part is usually writing the right tests for the spec.
<Zakim> manu, you wanted to commit to something
bigbluehat: We should be able to start with two or three easily.
<bigbluehat> +1
manu: +1 to bigbluehat. I want to commit us to something. Let's say you and Parth discuss some time this week, and then we share a minimal viable test suite with the group.
bigbluehat: This sounds very doable. Benjamin and Parth are signed up :)
bigbluehat: We can use existing DB infrastructure to test against. Would be good to have other implementations too.
<bigbluehat> w3c-ccg/
peacekeeper: We have a resolver up and running, happy to work with Benjamin and Parth to provide an initial endpoint/implementation.
Wip: I will confirm this on the calendar and talk to Pierre-Antoine about using the CCG setup.
peacekeeper: So we're meeting in 2 weeks?
Wip: DID WG call on CCG infra, I'll talk w/ PA on that.
manu: I can set up the CCG call time, let me know.
manu: I have to do some configuration about Google Meet regarding meeting minutes.
WIp: Cu all tomorrow.