DID WG Topical Call on DID Resolution contracts — Minutes

Date: 2020-06-18

See also the Agenda and the IRC Log

Attendees

Present: Dave Longley, Amy Guy, Brent Zundel, Markus Sabadello, Manu Sporny, Justin Richer, Orie Steele, Dmitri Zagidulin, Daniel Burnett

Regrets:

Guests:

Chair: Brent Zundel

Scribe(s): Amy Guy

Content:

Brent Zundel: anyone object to getting started? I’m not super comfortable with it but I think we have the primary discussers here

Manu Sporny: during the last call we had a productive back and forth between dave longely and justin around concerns related to the abstract vs concrete-ness of the PRs
… in the meantime both dave and markus took that as input
… the chairs and editors requested that dave and one of the editors work together to put a concrete PR together, which markus did
… markus has 2 PRs

Manu Sporny: https://github.com/w3c/did-core/pull/322

Manu Sporny: https://github.com/w3c/did-core/pull/323

Manu Sporny: and perhaps the best thing to do is start the call off with markus and dave taking us through their thoughts on those PRs
… how they address the concerns raised by the WG and address concerns raised by justin, and then follow that with discussion
… any objections to that approach?

Markus Sabadello: talk a little bit about these PRs..
… my recollection from the last call was that one of the main open questions was around the type or nature of the returned did document, of the resolution function
… some discussion whether the function should return the did document in the form of a byte stream in one of the conformant representations which could then be parsed and have a mime type and metadata
… or if the function should return a did document as an abstract data model, not one of the concrete representations
… the argument for that was that for resolving certain did methods such as did key you never actually need to work in any of the conformant representations
… on the other hand the argument for a byte stream and a concrete representation was that then a client or caller of the resolver would be able to parse it according to one of the known mime types and use content negotiation and all of that
… what i tried to come up with in these PRs was some resolution to this question
… the first PR, 322, that adds in addition to the resolve function also adds the dereference function
… the resolve function always returns the did doc in an abstract form
… the dereference function always returns the did document or some other content as a byte stream
… the idea behind this PR was it would have some kind of layering which we talked about in previous calls
… dave has talked about abstract resolve and concrete resolve
… this may be a little bit in the spirit of what dave said
… resolve returns an abstract did doc whereas dereference returns a did doc in a concrete representation
… the second PR 323, is another approach which does not talk about dereference
… ti only has resolve
… but it says that the resolve function can return either the did document in its abstract form, or as a byte stream of one of the concrete conformant representations, it allows both

Markus Sabadello: it depends on the input metadata whether an accept metadata property is given or not
… these are two ideas to address this
… i’d be curious to have some discussion to see if one of them works
… or if one of them can be improved to make it work

Justin Richer: markus, thank you for doing the work to put actual text down in these proposals
… i am very thankful for that work because I’m about to say I don’t like either proposal
… but for specific reasons we can discuss
… in 322, my issue is that it conflates in my view the functionality of the resolve function
… so that means that we now have to account for DID URL parameters and lots of other things
… I don’t believe that was really the intent of what was getting used here by these functions
… the second one, 323, I really have a problem of saying something could return one thing or another
… of very very different natures
… and there’s not a good way to differentiate what i might get back if I call this in any particular library
… what i had suggested last week was to add an abstract function separately that returns the did doc as an abstract data model
… and keep the definition of the existing function that returns a parseable data stream
… I understand you were trying to skate between those, but I disagree with how it’s sliced up
… I don’t care if we rename any of the functions, I don’t care what anything is called

Manu Sporny: perhaps what would address your concern is if we did have that layering
… I think 323 comes really close to addressing everyone’s concerns modulo what you just said
… my expectation is that if we said here’s an abstract thing and it returns an abstract thing and then we had something on top
… let’s call the first one abstract resolve and the second one resolve or concrete resolve
… and the concrete resolve returns a concrete byte stream
… you would be happy with that?

Justin Richer: yeah
… basically if there are two separate functions and the difference in functionality is clear that’s fine
… both of markus’ PRs try to take what used to be the resolve function and sort of mash it in with another function definition
… that leads to unnecessary ambiguity
… i think we would be best to avoid that

Dave Longley: I’m glad to see that we’re maybe converging on something, not sure what it is yet
… my question for justin is, the thought here was to layer infra in here
… so the return type for the did doc would be an ordered map
… the same thing would be used for the metadata that also comes back
… part of the trouble is finding consistency at the level of abstraction
… if there’s a problem with returning the did document as an ordered map I don’t understand what we’d return the metadata back as that it would not have the same problem
… if we use the infra spec and say it’s an ordered map, that gives you the behavioral information you need to know it’s a set of properties and values
… so you can use the rest of the spec in the abstract to understand what you’re working with

Markus Sabadello: the reference to the infra spec is definitely something that could be added
… the second PR that only has the resolve function which can return either abstract or a byte stream, that could be modelled with the infra spec
… it either returns the ordered map, what dave just said, or it could return the byte stream or byte sequence
… ia lso understand justin’s concern that this is violating layers or conflating things
… one question i have is if people want two different functions, concrete and abstract, then why is not resolve and dereference?

Manu Sporny: I want to point out things about the infra spec that people in the group might not know about
… the infra spec is kind of this boilerplate spec
… it’s something w3c specs can refer to normatively

Manu Sporny: Here the link to the Infra spec: https://infra.spec.whatwg.org/

Manu Sporny: it’s like a meta spec
… it collects boilerplate language that’s used across ietf and w3c specs over the decades
… it’s useful to us here where w’ere talking about abstract data structures that are intended to have concrete representations at some point
… I think the mistake the did core spec has made to date is not formalising that more
… another thing that folks might not know is that even though this is a whatwg spec there are w3c specs, lately json-ld 1.1, that used infra completely for its abstract data model concepts
… this is shared with many of the browser apis, restful and http apis, so we wouldn’t be the first by any stretch ot use it
… it would formalise the abstract language w’ere using in the spec such that it brings it in line with the rest of the web ecosystem
… when you’re talking about a data model
… with the combination of markus and dave’s PR, 323, and a commitment to move over to infra, we’ll get the abstract layer defined in a way that is well understood
… if we do a clear separation between that layer and the layer that justin is suggesting, the concrete layer, bytes and representations, that is something that sounds like it would work for the group
… and make the spec better in general
… that’s my suggestion is if we can get to saying 323 is what we’re going to do modulo the proper layering between abstract and concrete
… and we’re headed towards infra, that addresses everyone’s concerns
… anyone would object to that?

Justin Richer: to respond to dave, the problem is not returning the did doc as an ordered map, that is an appropriate data structure to use
… though it is slightly different from my proposal for metadata
… the problem is the conflation of it being either an ordered map or a parseable representation
… if we separate those into different functions then that’s fine
… that’s what i’ve been advocating for week snow
… is to separate those so the function definitions are clear for both implementers
… and more importantly for consumers of those functions
… the people who are going to be calling them need to have a reason for calling them
… if they’re going to be getting back an ordered map that conforms to the did document data model
… if they’re going to be getting back a stream in a conformant representation that is also great
… both are going to be required at different levels of interop
… i was not trying to define the former
… adding it to this Pr as an additional function is a fine addition and what i think we should do

Markus Sabadello: justin i agree it’s very useful to have this function that returns a parseable byte stream
… is it important for you that this function exists or is it also important that this function is called resolve? could we call it something else and could we call the abstract one resolve?
… could we layer it that way or is it important that the one that returns the parseable byte stream is called resolve and the other is called something else?

Justin Richer: dmitri just threw a suggestion in chat to call it resolveStream.. call it that!
… let’s call the new one resolve
… and define it separately even though it’ll take the same input maps and output maps
… the only difference is it will be the did document itself
… we should not put explicit ties to layering between them
… we need to stay away from the algorithm definition
… how you get there is your business

Jonathan Holt: i don’t want to overcomplicate it. Infra spec..

Dave Longley: +1 to resolve() and resolveStream() <– and a DID resolver MUST implement one of these.

Manu Sporny: next steps.. feel pretty concrete
… if there’s no objection we take 323, split things up into a resolve and a resolveStream, resolve is the abstract thing
… we will attempt to use infra on that
… there will be a separate thing which is resolveStream that gets us the bytes justin was going for

Justin Richer: suggestion that we don’t rely on CamelCaseFunctionNames for differentiation. :) so maybe resolve_stream or just resolve stream

Manu Sporny: epxected input and output is more or less the same so there will be some finagling around language
… to me that feels like .. if there are no objections markus will probably take the lead on that with the support of other editors
… hopefully in an async fashion

Markus Sabadello: this is a great proposal for a next step
… happy to work on this and collaborate

Dave Longley: +1 thanks, markus!

Manu Sporny: pausing to see if there are any objections

Justin Richer: let’s be mindful of the name
… this is silly but because it’s a two word name we need to be careful and deliberate about how we express it
… i don’t know what infra allows us to do
… i would hate to live in a world where i could have uppercase and lowercase resolve stream defined by another spec

Manu Sporny: +1 to that
… next item is the metadata discussion
… we kind of touched on it a bit

Dmitri Zagidulin: -1 to snake case, though, as that would mess with several language’s linters

Manu Sporny: the main point of disagreement there is maps of strings, what justin proposed, or a more extensible map like a map in the map etc
… going back to something a bit more high level
… i think the suggestion .. we’ve got the abstract stuff down fairly well.. infra has suggestions
… if you are using infra to define, they have an ordered map, and if you want to turn an ordered map into json you call toJson
… it maps very cleanly to programming languages that support json
… the base suggestion is all metadata is expressed as infra
… it sounds to me like the right data structure is an ordered map
… then the only thing we need to debate is what is allowed in that ordered map
… is it pairs of strings only or do we allow more complex data types?
… looking for objections, strong opinions

Dave Longley: +1 to reuse Infra, it brings formalism that helps implementers

Manu Sporny: hopefully before the debate about maps

Justin Richer: i think there should be a general objection to it being an ordered map
… i think an unordered map is going to be more widely serialisable in a way that you can count on it being serialised appropriately
… json as a core spec for example does not guarantee key order of any type
… you can kind of rely on document order but i’m sure as all the jsonld folks know you need to go above json to guarantee that
… an unordered map would be a better lowest common denominator
… i bet infra has something like that
… i’m interested in the discussion of what the content is and i can provide background as to why i proposed a map of strings
… i don’t know if we’re going there yet..

Manu Sporny: https://infra.spec.whatwg.org/#maps

Manu Sporny: to address the un/ordered thing
… they only have ordered maps
… there’s no such thing as an unordered map
… the way that a lot of the web platform is implemented, javascript and other languages, they did a survey and found that for consistency purposes there are certain languages that only support ordered maps and sets
… all of the languages that they could find perform some kind of ordering
… meaning that the lowest common denominator is an ordered map and an ordered set
… that is why they don’t provide the other option
… we can say we’re using an ordered map (infra) but order doesn’t matter
… that allows implementers to use an unordered map
… but even that language is frowned upon
… there can be modest implementation concerns that arise when someone goes oh i’m going ot use an unordered map

Markus Sabadello: for reference, justin_r ‘s PRs about metadata structures: https://github.com/w3c/did-core/pull/298, https://github.com/w3c/did-core/pull/299, https://github.com/w3c/did-core/pull/300

Manu Sporny: we can inject language, it’s viewed as an antipattern, but not one that would cause the TAG to object
… that’s the back story on unordered sets in infra

Justin Richer: i was going to suggest that if the only tool we have is an ordered map then we say order is not to be relied upon by consumers
… for the type of information we’re looking to get in and out that’s going to be universally true
… for all the examples i can think of
… for someone to have a dumb client that always takes the first key expecting it to be the error field and not doing any checks
… i’ve seen worse stuff in the wild
… so yeah, that’s fine

Orie Steele: See also: https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-16 regarding JSON and its lack of canonization support.

Manu Sporny: +1 to that
… so now let’s jump into the bikeshedding around what can go in these maps
… to frame the discussion, this is a general discussion around metadata
… we have a variety of resolution metadata, did document metadata
… we hope to express them all in the same way, using infra
… what goes into this ordered map where the order cannot be relied upon?
… justin can start with arguing for pairs of strings

Justin Richer: the core argument, started with a conversation markus and I had months ago, because these are the kinds of data structures we want to exist outside the did doc and its rich description capability
… we wanted something usable in a wide variety of different representations
… it should be useable in spaces where you don’t have a way to do an object within an object, or an array within one of these fields
… we shouldn’t force developers to invent that kind of structured serialisation
… i’m thinking specifically of smtp and http style headers
… please don’t jump on those protocols
… but that style of header where you have the header name and then the value of the header
… is just the rest of the string
… and you need to be able to encode whatever value you want
… inside that
… i can’t think of a single thing that we would want to encode in the metadata that would require for there to be a general purpose data structure style mechanism
… what we should have is something very limited
… when you say a date it is a string that represents a date in a specific format
… so i know exactly how to interpret this field

Markus Sabadello: confirm that justin and i and others
… talked about this in did resolution calls
… not going to define this in the did core spec
… somehow this metadata will be represented
… could be in a protocol specific way such as http headers
… what we’ve been calling a binding
… i agree with justin that the plain string key value pairs are simple and familiar to many developers
… that makes it applicable in many protocols
… one example that always comes to my mind where the more complex data structure might be useful is that of proofs
… we just removed proofs from the core properties
… however a proof might be something somebody would like to add as metadata about the did doc
… then people may way to use jws or linked data proofs
… those things may be come values
… if we say we have string keys and values on the contract level in these function definitions when we define the metadata data model, we have string key sand string values
… the values could still contain a serialised linked data proof or a serialised json web signature
… that’s then up to that specific property to define
… we would on the contract level we would define that
… I did want to mention the proof use case

Dave Longley: In response to talking about keeping things simple, I’m not against that, I don’t think we should define anything more complicated than what we need

Dave Longley: https://tools.ietf.org/html/draft-ietf-httpbis-header-structure-17

Dave Longley: the http wg is working through a number of mistakes that were made in the past that made things very complicated
… this includes defining ‘structured headers’ which is now called structured field values for http
… where they’re trying to deal with the litany of types of headers and encodings that people have made that are inconsistent and hard to parse
… this has become a headache as http evolved
… instead of just having simple strings and values, it’s defining lists, dicts and items for field values
… it was considered a mistake to allow lax parsing and different encodings in the past
… they’re thing to learn from mistakes they made, we should take that into account

Manu Sporny: my comment had more to do with what programming environment
… the expected programming environment in which the spec is going to be used
… i get that there are iot devices
… microcontrollers and things like that
… I’m struggling with whether or not those devices .. whether it’s going to hit those devices
… i’m struggling to think of a device that is going to need to process those metadata values
… iot is the only use case I can think of
… typically this processing happens at an iot hub
… even in the more modern devices
… 100s of mhz of processors, enormous amounts of memory, shipping on $1 devices
… i get the argument for lets do something simple
… and note the work being done in http and structured headers

Dave Longley: it may be that “something simple” actually ends up being “more complex” once people start defining their own metadata.

Manu Sporny: i’m struggling to think of a programming environment that wouldn’t be able to parse json

Orie Steele: you can flatten json and then use Tag Length value, and still support both hierarchical data, and string only interface.

Manu Sporny: my expectation that every programming environment we can think of can handle at least what you would need to put stuff into json

Dave Longley: because we didn’t hand people the primitives they need.

Dave Longley: (they will invent their own and invent different versions of it)

Manu Sporny: fundamentally that means we’re talking about a corner case

Justin Richer: Orie: that’s basically what I had in mind

Manu Sporny: a very isolated corner case
… if we don’t do this and we use strings then we go the route that http headers went down
… that’s not to pick on http, it’s just to pick on when you don’t define some of these data structures they end up creating their own syntaxes on their own and that leads to problems with interop

Orie Steele: 100% expect objects to get passed around though… nobody is going to “just use strings” :) they will use flatten, TLV, etc…

Manu Sporny: a failure scenario is where people are defining metadata in the did spec registries start inventing their own formats for maps and lists
… that could be a danger to interop even though the intentions were good in the beginning
… infra is kind of a baseline
… these are the data structures every programming language has
… I’m wondering why we need to constrain further than that

Dmitri Zagidulin: two things..
… repeat what manu said with regards to .. caution that simple key value pair is something that sounds simple that is going to require custom parsers for every implementer
… iw ant to avoid that. http headers are a nightmare
… What’s an example of a metadata value that isn’t content type?
… why do we have multiple metadata value strings

Justin Richer: from my view, i don’t think we have much right now and are basing a lot on conjecture
… that was a large part of the reason for constraining to data represented as a string
… i think people are getting hung up on its a string.. couldn’t you just do a flat json object? yeah absolutely
… i would expect something like that to be the guidance in the spec about how to represent common metadata types
… that alone doesn’t go far enough to represent a lot of really really common things
… best example is a timestamp
… you could use iso dates but those can be with or without timezones or fractional seconds
… there are about a half dozen common interpretations, most incorrect, of the isodate format
… or you could go the jwt route and use integer seconds since epic
… are those allowed to be fractional?
… we’re already going to need rules of interpretation beyond just the representation
… i think it’s a red herring that if we just say strings it’s going to be a disaster of everything
… i think we can provide guidance for common structures and formats
… i want to avoid having metadata processing required to do a full recursive tree parse of the metadata structure in order to get out a single value
… if i’m just looking for the content type and i don’t care about what’s int he proof i shouldn’t have to spend all of the resources to do this tree parse
… which could potentially have some deep value error
… i should be able to get this map, get out the value is care about, understand them unambiguously and move on with my day

Orie Steele: example pulled from the universal resolver today: https://gist.github.com/OR13/d68b75878e59c5f152118421e980b662

Markus Sabadello: responding to dmitri, some examples, the output of the resolution function contains did document metadata and did resolution metadata

Dmitri Zagidulin: wouldn’t that be in the payload though? (not the metadata object)?

Markus Sabadello: did resolution metadata is different between each process, would contain content type
… also information about the resolver
… how long it took, whether there was a cache, details about the resolution process that could be method specific
… did it have access to its own full node of a blockchain or some intermediary
… these kinds of things about the resolve and resolution process
… the other type is the did document metadata that is typically the same between different resolution processes
… only changes if the did doc changes
… such as created and updated
… what’s the version number
… my opinion this did document metadata could also contain things related to auth, controller, capabilities
… both types, the did doc metadata and resolution metadata, could potentially improve proofs
… did doc metadata could have proofs added by the controller
… did resolver could contain proofs added by the resolver
… there will be very diverse types of metadata
… hard to predict whether most will be very simple strings or whether we will see more complex data structures

Dmitri Zagidulin: thanks, that makes sense

Orie Steele: Microsoft appears to be planning to put pre rotation commitments to keys in the “did document” metadata, which are built using JCS…. a concept similar to KERI…

Dmitri Zagidulin: what’s the use case for putting all of those in the metadata object rather than in the second return value that is the payload
… if we’re aiming for the simplest possible structure, it would be a byte stream and the only thing needed is the content type
… what we’re doing is adding a second layer of indirection
… we’ve got this very complex .. given the type of things we want to express, this is a very complex arbitrarily flexible data structure
… if we’re going to stuff it in this metadata object we need a lot more specification
… we’re essentially duplicating all of the work in specifying the params needed to parse the payload, except we’re doing it twice with the metadata object

Manu Sporny: now we’ve had some discussion i wanted to get some feedback from the group with a straw poll
… we’re not making any kind of decision here
… no resolution, just get an idea from folks

Dmitri Zagidulin: Orie - oh, VERY cool, re msft’s pre-rotation commitments.

Manu Sporny: two proposals… one to use simple string key value pairs for metadata expression. The second proposal i’m struggling with, use string keys with any data types supported by infra..
… for values in metadata
… i could change infra, many of us are not familiar with it, i could say something like json compatible values

Dmitri Zagidulin: can we add a third proposal? for a single string ‘content-type’ value being the only metadata, and everything else going into the document?

Dmitri Zagidulin: the second return value of the resolve() function

Proposed resolution: Use simple string key-value pairs, where the keys are strings, and the values are strings, for metadata expression. (Manu Sporny)

Justin Richer: +1

Dmitri Zagidulin: -1 (not sure there is such a thing as “simple” key-value pair string representation)

Orie Steele: -1 i will expect to be handling lots of custom stringified objects… which i’d rather not

Manu Sporny: -1, feel like it will result in complex parsing rules for values (which happened for HTTP Headers)

Dave Longley: -1 seems too limited (and will result in people inventing their own mechanisms) BUT i want to find a way to support justin’s use case

Markus Sabadello: 0

Brent Zundel: 0

Justin Richer: my arguments for the string pairs are not defining the representation

Manu Sporny: infra unordered map is an abstract concept
… it does point to using infra to define this and allows values to be up to the level of complexity that is expressible as json

Proposed resolution: Use an Infra unordered map for metadata expression where the keys are simple strings and the values are compatible with Infra and serializable as JSON. (Manu Sporny)

Dave Longley: +1 this is the better choice (i think it will cause less suffering than Justin’s use case of not wanting to parse things you don’t need – BUT, i would still like to see if we can support Justin’s use case somehow)

Orie Steele: +1

Manu Sporny: +1

Dmitri Zagidulin: 0

Justin Richer: 0 because this is just doing string maps but defining a serialization

Brent Zundel: 0

Justin Richer: (and we still need to do things like date parsing)

Manu Sporny: need to discuss this more, on the next special topic call
… bye!