DID WG Topical Call on Resolution Contracts — Minutes

Date: 2020-05-28

See also the Agenda and the IRC Log

Attendees

Present: Daniel Burnett, Markus Sabadello, Justin Richer, Amy Guy, Dave Longley, Manu Sporny, Drummond Reed

Regrets:

Guests:

Chair: Daniel Burnett

Scribe(s): Amy Guy

Content:

Justin Richer: after comments on the PRs I sat down to make one final effort in trying to slice and dice these suggested changes for the DID Core Spec focussing on the resolution contract and metadata structures
… I did not touch the DID URL dereferecing stuff with the idea that joe still has a problem with that and it’s not a hill that i’m planning on doing much fighting or dying on
… what I did is made a bunch of edits and sliced them up into 7 different commits, made 6 PRs
… these PRs directly build on each other
… if you were confused as to why there are so many, the 1st is just this commit, the second is these 3, the third is these 4, then 5 then 6 and then the last is all 7 together
… I tried to arrange them and the text in ways that the individual requirements and lines are built on top of each other so the group can decide what part if any it wants to deal with

Justin Richer: https://github.com/w3c/did-core/pull/299/commits

Justin Richer: this is the commits ^
… the PRs as a whole are numbers 295-300
… 299 is the one that has all of the commits in
… 300 stops without an additional commit
… the individual pieces you can look at the commits and see this is really a separate piece of discsusion
… I made them as different PRs because I don’t know what part the group has a problem with, that’s why we’re here
… . I can go through the changes..

Manu Sporny: thanks for all this work justin
… chopping them up into smaller bits is going to help us
… I would like to start from the bottom and go up and see how far up the stack we can go

Orie Steele: yes, thanks for the structure, i found it very helpful, and only really have issues with the last 2ish

Manu Sporny: I believe 295 is the first base one

Markus Sabadello: i think this is great work. Personally I agree with the content mostly. I think there were a number of open questions after the last special topic call
… we could go through them, or just do one PR after another
… one of the open question was does the resolution function return a DID doc in its abstract form, or a stream of bytes?
… another was about error codes and testability
… not sure how to do this, could talk about some of these open questions or just go over the PRs

Orie Steele: i think those questions will be addressed if we go in order

Justin Richer: these issues are addressed in separate commits
… I’ll go through the commits not the PRs so we don’t lose the context
… from there as we go up that stack of commits we’ll figure out where people start to have heart attacks

Manu Sporny: let’s not have an open discussion, let’s get concrete and process these PRs, they’re in an order that the questions will come up and be addressed as we go through the stack

Orie Steele: +1 no more open discussion lets review the commits, justin has done a great job of structuring this

Orie Steele: :)

Markus Sabadello: +1 to go through commits in order

Justin Richer: I’m going through the commits, we can back this into the relevant PR as necessary
… First off, the first commit says all resolvers must do resolution
… hopefully that is not controversial
… for one method, and return the document in a conformant representation
… this cements the fact that the document being returned is in a representation for this
… this is a definition of what this resolve function does, without putting requirement onto exactly what the types are

Markus Sabadello: I suspect there could already be some disagreement on that first paragraph where it says in at least one conformant representation

Justin Richer: I think I understand where you’re going and I’m going to ask you to hold that thought until the next one where we get into data types or conformance
… I think that will be more pertinent there

Dmitri Zagidulin: I want to ask about the signature of the resolve function
… right now pr 299 says the response is did resolution metadata, did document stream and did document metadata - so far so good - how do we know what format the did resolution metadata is in?

Justin Richer: as of this commit we have no clue.

Dmitri Zagidulin: I think that won’t work

Justin Richer: I agree. This commit on its own is useless. We’ll start here and add stuff on top of it

Dmitri Zagidulin: in the fetch spec it has two return values, the content type and the byte stream. the content type is separate because it is needed to interpret the byte stream, but here it’s not possible ot interpret it

Orie Steele: maybe we can go through everything and then take comments?

Justin Richer: I agree

Dmitri Zagidulin: why would we pull in a useless PR

Justin Richer: because the others will add onto it
… waves mouse around a screen to do tour of commits

Dmitri Zagidulin: I’m specifically looking at the final product of all of the commits and that does not address my concern

Justin Richer: content type is returned as part of did resolution metadata, which is defined as a string map data structure (not json or cbor), it is a data structure not a serialization format
… the document itself remains serialised and therefore you can do that
… the portion of the fetch spec you’re referring to, the body mixin, that is an optimisation on top of fetch
… where it pulls in content type from the headers object of the http response object and pulls in the http response entity as the body
… so you can get all of that stuff without the body mixin
… the body mixin just makes it easier, an optimization for a common usage pattern
… I’m not looking at optimizations of these functions, but the raw parts that we could build optimizations from

Orie Steele: +1 to everything justin is saying :)

Manu Sporny: going through everything will raise a bunch of questions… some foundational things I’m concerned about

Daniel Burnett: we can have larger discussions further through..

Manu Sporny: two things.. I feel like, dmitri usually I think groups, when we get into a mode we’ve been in, we’re frustrated and not making a lot of progress, there are objections and so many different thoughts… sometimes it helps to break it into little pieces, understanding that what we’re doing is making incremental improvements to the spec that by themselves dont’ really do anything but once they’re stacked together it is the full solution
… I think we’re doing the right thing by processing these one at a time

Orie Steele: ask yourself at each stage, will this prevent us from getting a thing which I believe we need… not does this thing do what we need right now…

Manu Sporny: there are two concerns i have with 295.. function.. I’d like the group to settle on language
… we’ve been saying contract and function and process, some have been using it interchangeably or a definition is not shared
… what are we documenting here? is it a process or a function? or both? definitely not an algorithm
… I’d like to settle on that language and have it ripple up
… the other concern I have is around the normative nature of the very last statement on 277
… it says all conformant implementations must implement a function in the following form. It is unclear what test, or how that changes an implementation
… is conformance on this statement going to be self reported?
… or are we doing something testable?
… I really prefer us to have testable statements, but need to understand what your intent was in this language
… at that point I feel like we can merge this PR if we have consensus on these two items

Justin Richer: in my opinion it’s not worth bikeshedding the word function right now

Dave Longley: and there’s perhaps some conflation around how this function is described… and even after all of the commits for all the PRs are considered, the “function” is still left somewhere in the middle between an abstract mathematically function and function in a concrete interface language whereby the “function signature MUST not change”

Justin Richer: in terms of the normative statement, the testability there, as I have said, what we are looking at testing is not implementations themselves but allowing a callable version of this to be tested
… we’re going to provide tests for callers of this function
… in order to make this testable we would implement a mocked version of the resolve function which takes in these inputs and gives these outputs so people can run their stuff against it and tell us how to make their thing call the resolve function, or we provide them with a scriptable way to do that
… collecting the results is about instrumentation

Orie Steele: if we don’t use a mocked version, we will be implementing tests for a specific did method… which will invite objections, bias, etc…

Justin Richer: I believe you think the semantics around the word function are unclear so it’s imprecise for some people and too precise for other people. You prefer process. I think you believe that by settling on precise language for this right now it will make clear what we’re trying to do

Manu Sporny: only the first quarter of what you said was accurate
… I want to get consensus on exactly what we’re defining
… if it’s a function what is the definition of function? abstract/concrete? if it’s process then the way we’re writing that code block at the bottom changes a bit
… the other concern I have is that creating a mock function that’s called, taking my editor hat off and putting my digital bazaar hat on, feels like a waste of time
… it is writing code that I don’t know where we’d integrate it
… I would much rather spend time actually putting our resolvers against test suites, rather than writing code we’re going to throw away
… if it’s a function we actually call, are we all writing it in javascript or python, how is that implementation going to happen? I don’t understand the usefulness of writing a test suite that does that

Justin Richer: you don’t understand the usefulness of mocked functions?

Manu Sporny: How do external processes call those functions?

Orie Steele: I prefer function over process, I find process system-oriented
… Regarding the points about test suite, there’s a tremendous amount of value in mocked lower level interfaces
… I’m guessing the resistance to that is people want to see test suites around the body mixin version of this that you’d see in fetch
… we can only get to that if we can get through tests for mocked versions of the function
… we can’t get to tests that apply to a specific did method
… people implementing a did method they’ll implement the mocked version as a way to see if their method conforms
… it’s valuable as a developer
… a lot of pushback is conflating with the higher level commits that come later

Justin Richer: the mathematical, lambda calculus definition of function is what i was after

Dave Longley: when we talk about having a mathematical abstract dfn of function, that’s what i can get behind

Orie Steele: +1 to church turing thesis / haskell style function signature

Dave Longley: it seems like when we look at these commits is what we’re trying to do is doing those two things at once, neither one clean
… we’re stuck in the middle between a mathematical function and a concrete interface language with a function signature that must not change
… I have written actual resolvers and this function would not match in a concrete sense
… I could squint at this and say it matches in some abstract sense
… I’d prefer us to say we’re defining a mathematical function with inputs and outputs
… when we start talking about types and doing other stuff we’re in between, not going all the way
… if we did it in webidl we could write software against it, actually write resolvers that could do that and be tested, which is fundamentally different that we have abstract functions and we self report that we’re doing the right thing
… in the middle is very confusing

Markus Sabadello: I was also using the term function in my original PR to try to emphasise what dave just said
… it’s just the interface with inputs and outputs, without algorithm, with out implementation of that function

Drummond Reed: +1 to using the term “function”.

Manu Sporny: It sounds like we’re getting consensus around the use of function in the abstract mathematical sense. I think the PR should make it clear
… I don’t know exactly where we put it but it should be somewhere so it’s clear this isn’t a concrete anything that’s being mentioned, that it’s abstract in every sense
… I think that resolves the bikeshedding

Drummond Reed: +1 to “function” being in the abstract mathematical sense.

Dave Longley: +1 to this being an abstract function, can support that

Manu Sporny: it sounds like there’s support for mocked stuff in the test suite, if there are no objections to that we can move on as well

Dave Longley: that was super helpful!

Dave Longley: +1 for progress

Justin Richer: this one (which?) adds types to all of the inputs and outputs

Manu Sporny: https://github.com/w3c/did-core/pull/296/files

Justin Richer: these are not necessarily meant to be precisely what the final type of everything is, but I wanted to have a stake in the ground to say this is a thing it could be and we can figure out if we need to adjust the language around this
… something dan says often which I believe in, the goal of this is not to necessarily change everything and come up with the perfect solution now, but to make things better
… what I’ve said is that the input vars - I agree this is very programmer-y engineering language but bear with me - are two variables, one is a DID and it’s a conformant DID as a single string, not a parsed data structure, just a string
… it is always required
… the other thing is a metadata structure map which we haven’t defined yet

Dmitri Zagidulin: why string and not a byte array?

Justin Richer: this is the input option to the resolve function in addition to the DID itself
… (strings are byte arrays)
… the section that talks about input options which are a different part of the document, it’s clear this are external to the DID itself, but the existing language doesn’t have a place to hang anything. This is trying to give this section a place. There would be additional changes to the document as a result of this which have not been made in this PR

Dave Longley: strings are not byte arrays, they are arrays of characters (and if we get concrete, this is important, if we stay abstract perhaps not so)

Justin Richer: the first output, I did shuffle things around, the order from what markus and I had originally wrote, and put the resolution metadata first, because this is always required in all circumstances
… I don’t say what goes in it in this PR, that comes later
… for a positive or negative resolution result we’ll always have did resolution metadata
… the did document string, my intent for this function that this is something that happens pre-parsing, pre-validation, before you know what format the did document is in
… if you were working at higher level and you were working with the parsed, abstract doctree, this function is not for you. more on that in the next commit set
… this says that the document must be a byte stream and we could define it as a buffer or a number of different things to get the point across. byte stream is relatively understandable. I dont’ care as long as it is a set of things to throw into a parser
… we then say nothing is requiring you to parse it but we assume you will. the result of the parsing will be an abstract data model, which can be processed as linked in the document

Dave Longley: I’m worried about the level of specificity here, flowing between the abstract and concrete
… talking about something as a string is abstract, as opposed to talking about something as a byte array
… we have resolvers people have already written, that we’re trying to make sure are interoperable
… important to look at the existing implementations to make sure whatever we’re defining as a resolve function - the lowest level primitive - we need to make sure those implementations implement this function and use it
… if we’re going to say this function is not for you - a significant number - we’re creating a work of fiction because it’s not for the resolvers
… we need to make sure whatever primitives we come up with work for these existing systems, or it makes sense to them to match the spec

Markus Sabadello: I think that’s a topic we have at the end of the last call whether the value you get back is a stream or a serialized did document in one of the conformant representatons
… that’s what I was thinking in my original PR, then we have input options related to the content type
… on the last call dave made a valid point that you might resolve a did and there’s no parsing or sterilizing of a did document byte stream
… did key, you resolve that and you should be able to execute an implementation of this resolve function on the did:key did and get back a did document, but there’s not any byte stream
… I tried to think about this today, I’m not sure if it’s easy to define a clear layering of one thing on top of the other

Dave Longley: is the function abstract or not? (we just agreed that it is, so why are getting so concrete with how to represent the abstract data model flowing in/out of it?)

Markus Sabadello: in some cases you want to resolve a did and get back the did document in one of the representations
… in some cases you want to get back the abstract data model
… I wonder if the function signature should allow both

Dave Longley: if we want to get concrete that’s fine, BUT, it has to work with real implementations.

Justin Richer: it is abstract but typed

Justin Richer: I think you’re asking about a different function, which is okay and also valuable

Orie Steele: https://gist.github.com/OR13/edb40f24a4fffc3bb1a63488163142dc

Orie Steele: ^ this is what they are asking for

Manu Sporny: I had the same general confusion in reading this
… we are saying it’s abstract and getting more and more concrete
… the nuance is you can have something abstract but be very specific about the expectations around the abstract interface without getting to bytes
… I’m wondering if at the end of the day we’re still going to be left with an abstract mathematical function
… if the answer is yes that’s fine and would probably work for some of the other people
… it does raise questions about how we test that
… I’m putting a pin in that for later
… as far as the PR is concerned, assuming we’re talking about an abstract thing but getting more specific about the typing, the only part of the PR I felt probably didn’t belong is the very last paragraph where we say things like did resolver implementations must not alter the signature of this function

Justin Richer: that’s in the next section, jumping ahead

Justin Richer: https://github.com/w3c/did-core/pull/299/commits/7169f00f909948b221d1e4b619012dd1eae316a1

Dave Longley: if we’re going to define an abstract function the inputs and outputs should have abstract types, too
… the other concern I have is if we’re going to say this resolve function is not for resolvers let’s name it a different function

Orie Steele: I call it “resolveBasic” see the gist

Justin Richer: fine with renaming
… I didn’t add the additional functions to this to keep the conversation simple
… in terms of abstract data types I am fine if we have a better way to say I’ve got a whole bunch of stuff that I don’t know what it is but I can put it into a parser
… if there’s a better way to say it I’m fine with that

Daniel Burnett: the conversations are good, the first point about function is one of the key points, we’re making progress on the level of specificity

Justin Richer: this next point, what manu was commenting on, all of this is trying to say is you don’t get to redefine this function which I have named resolve but might be different, but you don’t get to redefine it with an extra parameter or extra output
… you can define other functions to do that

Dave Longley: Note: the JSON-LD API spec uses an “internal representation” to talk about things in the asbtract so as to not confuse them with a particular concrete representation: https://www.w3.org/TR/json-ld11-api/#dfn-internal-representation

Justin Richer: but you’re not allowed as an implementation to redefine what this function means, is what this text is supposed ot say

Manu Sporny: I get that. previously we said you have to implement something of this structure. this feels like the counter of that, restating something in the negative
… you say you must do this, and later on you say you must not change the thing you must do
… it’s redundant, let’s take it out

Orie Steele: sounds like an editorial comment, that we can clean up later.

Justin Richer: no, it’s better to be more specific

Orie Steele: if its redundant, its easy to remove.

Dave Longley: It uses that language when talking about algorithms (whether or not we want to talk about “algorithms” here), just pointing out that we’re still mixing “concrete” and “abstract” when we start talking about needing to “parse” something – that something is in a concrete representation.

Manu Sporny: there’s a disagreement there around adding normative statements that duplicate other normative statements that confuses people, it raises the question, someone reading hte spec would say they’re asking me to do this thing, now they’re asking me to do it again, I must be missing something
… we can get input from the rest of the group, but typically when I’ve been editing specs, I try to avoid restating normative statements in the negative because it raises questions
… there are multiple statements, let’s not debate every single one, but we need to figure out what we’re doing in this spec

Daniel Burnett: to respond to manu’s comment, I’m going to say we don’t have to be perfect right now.. while I understand the point you’re making, in the end we may decide to go through the spec with an eye towards reducing redundancy, I don’t know if it’s a problem if we do it now.. May be one of those things we can agree to do now and see later whether it needs to be removed
… one of the reasons for removal is not for people reading, but it’s easy to make a mistake if you make the same statement in multiple places, when you’re making an edit in multiple places.. tha’ts an editorial job later
… the only normative pieces is figuring out whether the statements are the same or different

Justin Richer: all of this can be changed after acceptance

Orie Steele: +1

Justin Richer: On this one, this defines property values
… I refactored how this text was working and took the input metadata properties, resolution metadata properties and document metadata properties, and instead of inlining them they have subsectons
… the values ar enot important right now
… this is just me stabbing something into the ground so there is something here
… one input parameter that people have talked about is talking about this is the format that I would like it in
… in terms of output, 2: content type if it’s a positive response, and error if it’s a negative response
… within error, values defined

Markus Sabadello: how I imagined it with my original PR
… however this assumes you get the did doc in one of the conformant representations as a stream, which may not be consistent with how you can think about resolution, maybe you don’t need it in any of the representations
… something to discuss, but in general this is the same

Justin Richer: thanks for bringing that up, that raises the question of what kind of interoperability this group actually wants
… I’m not convinced we know
… I am trying to define it along the lines of what I saw coming from different aspects of the group - take in a cbor document from someone else and be able to parse it and understand it and produce a json document, that level of stuff
… fi the group doesn’t actually care about that level of interop then I agree that none of this makes sense but there are enough parts of the group that do care that this is a worthwhile exercise

Dave Longley: one of the issues that might be leading to misunderstanding is I don’t think this abstract function I keep referencing is something that would be called or used after parsing, but internally
… this resolve function would take some kind of concrete inputs, call this abstract function, and then it would convert ot a concrete representation as an output
… I want to be clear that this abstract concept is something that would happen internally to the function we’re talking about. Not after parsing, not an extra function that handles the abstract thing
… how to represent what happens internally in an abstract sense

Justin Richer: that is defined as the read method on a resolver

Dave Longley: when you say that is..

Justin Richer: the read method, back a couple of levels..
… by using the read operation of the applicable did method

Orie Steele: Here is an example of did key resolution… https://gist.github.com/OR13/8a7e4d64629a3b3fb68758b672b052a2

Dave Longley: okay so maybe the problem I’m having is the way we’re talking about resolve is how we should be talking about read
… there’s a DID coming in and some abstract representation and the result of read is an abstract representation of the did document

Justin Richer: yeah that’s also why I keep seeing it as a higher level function
… i could have called resolve and parsed and validate and then hand back the abstract data model

Dave Longley: but in that process you had to deal with the abstract function at some point. that part is internal to that process, not external after you’ve called these other functions

Justin Richer: it depends on what level of interop you’re looking at
… if you don’t want to interoperate at the point of being able to read and parse somebody else’s representation of a did document, then it is internal

Dave Longley: we do want to be able to do that level of interop. in the spec we should define an abstract function that does resolution. then what you can do with that output is convert it into a concrete representation and pass it off to be parsed
… but it should be done as something, we should express that as being independent of the abstract resolution function
… the abstract function operates on abstract types, and people can wave how it works in their implementation, but it’s abstract in something goes in something goes out
… if you want to insert it into a concrete function, that’s optional
… if we talk about it that way things can make sense
… if we don’t talk about it that way, this abstract thing doesn’t exist and you’re alway soging from a concrete representation as if there’s a map and shoot it out of your resolver, that does not cover actula implementations

Manu Sporny: some of the text around unauthorized..

Justin Richer: can we put a pin in that? I picked these potential errors out of thing air, I have no stake in any of them, I suggest we move on

Manu Sporny: the second item is aligned with what dave was saying. It feels to me like we do want some things here very concrete
… my expectation was that because we have a data model spec we can be and probably desire to be pretty concrete about the inputs and outputs and what you can do with the outputs
… it’s still fuzzy in my head about how exactly that’s said
… I think there’s a missing link here between the abstract resolver thing and how that gets concrete
… I know where was text around bindings and that kind of things which felt a little too much of the did resolution spec getting into this did core spec
… I think one of the things I’m not seeing yet is this taking of the abstract model and abstract resolution function and turning it concrete without making the language complicated

Justin Richer: my attempt at making it concrete was to say it’s a serialized representation
… a function that returns an abstract data model is .. I don’t know if that’s even testable
… I have to drop right now, sorry

Dave Longley: a valid resolver should implement “the abstract function” (which should perhaps be called “resolve”) – and we can define concrete functions (similar to what “resolve” looks like in these commits) that may use that abstract function internally.

Orie Steele: I provided some gists, which show how this internal resolve is related to what’s being asked for

Justin Richer: apologies, I have a client meeting right now that I’m running, I couldn’t stay on

Daniel Burnett: Thanks Justin.

Orie Steele: if you’re having trouble imagining how these things are related, consider taking a did method you understand with support for resolution and implement a resolver interface that’s the same but happens to use this internally for no reason other than to make it concrete that you use json
… you can construct the resolve method you’re looking for with this internal representation and never see it externally
… for people who have only represented their document in one single representation this is not valuable because they don’t care to expose this internal function

Dave Longley: -1 to what Orie is saying, that’s the whole problem with it

Manu Sporny: yes, exactly

Manu Sporny: -1 – it’s a no op

Orie Steele: see if you can get resolve methods today to use this thing internally. Take your json object, convert it to a byte string, provide did method resolution properties to say it’s json

Manu Sporny: why are we defining it (and btw, fine w/ doing it because people seem to want to do it)

Daniel Burnett: obviously we need more discussion
… thanks everyone

Manu Sporny: thanks amy!