DID WG Topical Call on DID resolution contracts and metadata

Date: 2020-06-11

Attendees

Present: Justin Richer, Dave Longley, Amy Guy, Brent Zundel, Drummond Reed, Jonathan Holt, Daniel Burnett, Ganesh Annan

Regrets:

Guests:

Chair: Brent Zundel

Scribe(s): Manu Sporny, Amy Guy

Content:

Brent Zundel: Let’s get started…

Manu Sporny: We ended the last call with some people having epiphanies, so reviewing the thoughts on the way things are structured might be good
… has anything changed in the last week
… we’ll note that justin’s PR 296 was merged
… that’s the abstract function changes to justin’s PR, has been merged
… which means that now we need to consider the other PRs that layer on top of it
… reviewing the PRs that are layered on top to see if they change in any way no we’re talking about an abstract function, if that is something.. we need to talk about that as a group
… then effectively hand it over to justin to talk through his PRs
… would anybody object to proceeding in that way?

Markus Sabadello: I wanted to say quickly…
… Good to merge that
… Progress with 296 – there may be one issue, one comment on that other current language still that we discussed last time. Whether function returns DID Document in concrete representation in byte stream as opposed to function returning a DID Document in abstract data model.
… That’s fine, could address that later.
… I agree w/ proceeding w/ Justin’s PRs.

Manu Sporny: ..is it an abstract thing or a concrete thing, where’s the line

Dave Longley: Something that was useful that came out of the conversation…
… was that we want to talk about abstract function, that should also include things like byte streams, if we have abstract function about behavior of thing that goes in, behavior of thing that comes out, that can sit inside more concrete function, output things in concrete representation.
… if we model everything that way, we can have resolvers that implement abstract function, then concrete functions that can provide interop for people that want to receive things in concrete representation.

Justin Richer: It’s a point well made, I will say that that was not the extend of the intent of my proposal.
… my proposal was very specifically to get at the problem of the fact that we have various concrete representations and no way to differentiate between data in those representations and data about them.
… That may become more clear why I think a function define things in terms that could be handed to a parser.
… That is the function I was trying to define here, to be clear.

Dave Longley: I think it’s fine that such a function exists, but the abstract function should be in that function. There is confusion about what abstract meant… parsing didn’t have anything to do with my point, in particular.
… When I used the term “abstract”, what you are stating isn’t mathematical sense of abstract… logical data model… abstract function lives inside of concrete function. Abstractly implements mathematical function. Outputting a byte stream that can be parsed, that happens after this abstract function is run.

Justin Richer: I understand what you’re saying, Dave – disagreement on composition of functions, implementation of use case. Fundamentally, you and I are talking about two different functions. That’s why I didn’t have abstract in function definition that I proposed. The Editors requested that I merge that language. Take that as you will.
… I understand what you’re talking about, I think you and I are talking about a different function. They can be composed in a different way, in that if I have something that can give me an in memory abstract data model, I have called something under the hood that gives it to me in a concrete data stream that I parsed into that data model.

Dave Longley: parse_fn(concrete_fn(abstract_fn()))

Markus Sabadello: There is not a need to have a DID Document in any concrete representations, made one comment to PR that was merged, two functions that Dave mentioned - one returns abstract, one returns concrete… maybe that’s just resolve and one the other is dereference. We could make a choice to say resolve returns abstract document, which would mean changing what we just merged.

Dave Longley: The difference I’m trying to highlight - when I say abstract, I’m talking about conceptual data model, that’s another type of concrete representation.
… So, I put into IRC what I’m talking about…
… That’s how I see it working… similar in how other specifications are written, how to transform data from A to B, but only talk about data in conceptual way.
… It says nothing about output of conceptual algorithm.
… Implementations very well might output in a byte stream, then output in abstract data model… both of those things are not mathematical abstractions… where we’re talking about conceptual behavior. If you mean “abstract data model”, but that’s not what I mean by abstract mathematical function.
… We could take the JSON-LD Algorithms specification, another WHATWG specification - objects, arrays, things that are arriving and leaving, any resolver… like DID:key can be written against abstract function, is conformant with the spec. It’ll implement abstract function, might not implement concrete representation, serialized in a bytestream.
… SOme implementations might implement that, others shouldn’t, we should say you’re not a DID Resolver if you don’t implement that function, some don’t need to implement that function.
… We’re literally talking about different layers where abstract refers to mathematical sense, vs. parsed into a data model.

Manu Sporny: I thought on the last call we agreed to talk about some of this in the abstract mathematical sense
… the rest of the specification does talk in fairly abstract terms around data model and then we have representations
… that’s an example of talking about things in a fairly abstract way
… we do make them concrete at some point through the representations or assertions about things being URIs
… you can see what byte values result when you do that
… I’m concerned in this section if we muddy things together and don’t have that clean separation
… I don’t know yet if we’re talking about having clean layers or pushing all of these things together
… i understand where justin wants to get, that’s a good goal
… we should get there
… i also understand what dave longley is saying about making sure we get the layering correct
… because there are some implementations like did:key who will implement some layers but not the layer on top because it doesn’t make sense for them to do that

Dave Longley: i think english is largely failing us here … as “abstract” is not “concrete” enough :P … there are always many layers of abstraction to anything; there’s a way to talk about resolution in terms of behavior and concepts and there’s a way to talk about creating “more” concrete functions that implement that abstract behavior

Manu Sporny: If we’re getting into byte streams and concrete values here, w3c does have webidl
… they have an interface definition language for when you get concrete and you want people to implement things in a very specific way
… my hope was that we would be able to stay away from asserting webidl things
… you use webidl when you’re talking about in-browser apis and software libraries
… my hope was we wouldn’t go to that degree in the core spec
… it’s a decision the group needs to make by consensus
… those are some of things i’m seeing coming out of the conversation
… we’re not going to make progress until we get some kind of consensus around how we’re layering these things
… specifically it can’t just be a conversation between justin and dave longley, other people are going to need to weigh in on this topic for us to make progress
… it’s fine for everyone to explain their position, but once we get to the point where the positions are well known the group is going to have weigh in so we have consensus
… perhaps if folks want to suggest how we could reconcile justin’s position and dave’s position in a concrete way
… through changing spec text, that might be a good place to start to shift the discussion

Dave Longley: Right, so I don’t think Justin and I disagree that much, I’m saying there is yet another abstraction layer, where we only talk about data model and concrete function wraps that. If we were to change the language to not talk about specific representation, talk about logical behaviors and concepts, function to take one thing one way and convert it to another thing – define that behavior and that defines what resolution is.
… once we do that, if we implement that as a resolver - in particular, it would output a byte stream that could be parsed. If we do it that way, it enables implementations to implement some abstract function.
… that may or may not be useful… then we don’t even need to talk about parsing, that comes after. That’s how I see we could have a resolution.

Jonathan Holt: Perhaps we can go through a concrete example

Drummond Reed: What Dave is saying makes sense to me.

Dave Longley: an abstract “resolve” function defines the behavior of resolving a DID to a set of outputs that each have their own behaviors and concepts

Jonathan Holt: If we can go through a concrete example… this is too abstract, go through example, get DID Document… in the end, it will be a byte stream… in the start it is a byte stream… how we interpret the byte stream in certain formats, how it is performed to get to concrete serialization formats, to do authn, helpful to me to get specific example of that flow.

Dave Longley: a concrete “resolve” function would do MORE than that and say more concretely about what the outputs would look like, for example, they may be byte streams

Justin Richer: I’d like to propose to Dave that once again, you’re asking for a different additional function, put in a PR that defines that PR that defines that function, instead of what I defined, we can get back to function you defined in first place.

Markus Sabadello: I just wanted to see if I understand what Dave is saying correctly - we merged #296 before #295 - difference is #295… says what are inputs/outputs, doesn’t say inputs/outputs… if we only merged 295, if we specified names of input/output parameters… that is the kind of abstraction that Dave means?

Dave Longley: I’m looking over 296, the only difference between function I’m talking about is not mentioning byte streams… we didn’t talk about byte streams on DID Document metadata, abstract function outputs byte stream, concrete representation , DID resolution metadata, not a byte stream… concept and set of behaviors, DID Document is more concrete.
… The only difference is only NOT talk about it as a byte stream, but talk about it as a DID Document.
… This is why I think we’re conflating things, we are talking about things in abstract and mixing non-abstract things in, like byte streams.
… An abstract function, as an output, we can output a bytestream… but first is abstract, second is concrete, if we don’t do that, we have something that’s inconsistent.

Justin Richer: Glad you brought that up, thing I haven’t been able to talk about on these calls, is that what that metadata looks like.
… That’s one of the later layers, we haven’t even started to talk about that – #296 is incomplete and inconsistent, surprised that Editor’s wanted to merge it w/o additional discussion, but was also surprised that people wanted it sliced up in the first place. Doesn’t make sense yet, that’s why we’re having these calls.

Dave Longley: a simple fix is to remove “byte stream” and it will become consistent

Jonathan Holt: Ultimately, it’s bytes over the wire - byte stream is not a concrete representation, it’s just bytes to start off with.

Dave Longley: Yeah, this is precisely the problem, it’s not just bytes over the wire, that’s making an assumption of how this happens… that’s not the case for did:key - if we talk about it in the abstract, in the DID Core spec, we talk about this, abstractly, this is the behavior of resolution. The specific input is a DID, concept, the output are metadata, again, conceptual, we have an abstract function.
… The only thing we need to remove is remove talk about byte streams, if we do that, the function is abstract resolve and it’s consistent.
… then we can talk about concrete representations, if we take that approach,e everything would become clear and consistent.

Justin Richer: Huge -1 to taking that approach, a better approach would be to define completely abstract function as a separate function, redefine resolve function, implement as function that Dave is talking about, whole point about this is to provide something for interop at data level. DID that’s passed in isn’t a concept, it’s a string.
… I do not agree that’s the right approach, but do think what we can do can address Dave’s concerns - I believe they’ll address Dave’s point?

Dave Longley: A string is a concept, that’s not the point, the point is bytes over the wire - we’d be talking about UTF-8 bytes… but we’re not doing that.
… The only thing we need to do is just remove talk about bytes, then we can add concrete function after that is done.

Drummond Reed: I want to let Justin answer what Dave is saying, then go on the queue

Dave Longley: The only change we need to do is remove byte stuff.
… That’s an easy way to transition from what we have to what we need next.

Manu Sporny: justin says that one of the goals of these PRs was to get something concrete and testable so we’re not hand-waving over what the inputs and outputs are
… I believe we have consensus on that goal
… that’s one of the points I was making and didn’t want to put anything into this spec that was not easily testable
… no pushback on that goal, it still stands
… the other thing I’m hearing dave longley say is that let’s just get the layering right here
… assuming that goal, can we build the text in the spec up so we achieve that goal?
… right now it seems like we’re rushing into the goal and talking about things like byte streams and what we should have is proper layering here where we talk about this stuff in the abstract first at the bottom of this layer
… that will fall in line with the way the rest of the spec is written
… then we layer justin’s more concrete bytes on top to achieve the concrete representation that can be tested
… if we take the same strategy with the metadata discussion, abstract then concrete representations, that means the entire spec is talking about the things in the same kinds of layers
… in the abstract to start out, then we layer representations on top of that
… typically the test suite is written against the more concrete stuff
… from an editors perspective I know what language I would write to resolve this, but I don’t know if that would lead to objects to justin or anyone else in the group
… it is clear that people don’t care what the functions are named
… I don’t think that’s where the .. we will probably bikeshed the names, but the biggest thing here is layering and testability when we’re done
… I’d like to hear from justin if you feel like what I just said would be an appropriate way forward
… and if so you doing the PR or you and dave, or the editors talking it on
… I’d like to hear from you how you feel we can move on
… and get some concrete text in the spec

Justin Richer: First, I wanted to point out that the specific reason that I describe the input data as a string, a DID has it’s own data model, there is a method, there are method params, one could arguably say that this function takes in a data model that represents all of that… that’s not level of interop I wanted to define here, I wanted to define here a string that I could parse and interpret, it’s becoming very clear to me that part of this group

Manu Sporny: wants a different function, and that’s not what I wrote.

Justin Richer: I suggested that that function be added separately.
… If someone wants to take this PR and chop it up, fine.
… That’s not what I was trying to do, I was trying to solve a different problem.

Drummond Reed: As someone that hasn’t been involved in these PRs, I’m listening to this - want to second what Manu was suggesting, I am agreeing that there are DID Methods that don’t produce a concrete representation as a part of resolution, I know that’s why Dave wants it defined at that level, that’s not in opposition to what Justin has written. It’s an additional point about the content that needs to be incorporated. Now the question is, how is that

Manu Sporny: done editorially?

Drummond Reed: I’m willing to help work on a PR that brings all of this together.

Drummond Reed: In the case of did:key:, unless I’ve missed something, there is not other input than the DID.

Dave Longley: -1 to hand waving +1 to defining layers

Jonathan Holt: Certainly, I can see Dave’s point, and BTCR - you’re starting off, piecing things together, not familiar with did:key, but you’re creating the DID Document from artifacts, piecing things together, own abstracted layer, piecing things together, struggling w/ JSON-LD API Algorithms that are handwavy approaches, but it’s too abstract to me, as an implementer, having my code validate, not solidly formatted as I’d like… implementation

Manu Sporny: burden to handwave. More specific approaches needed, appreciate the need for arrangement, get these artifacts into state into common state.

Dave Longley: i think Justin has done a great job with the function he’s written – and it (incidentally) is pretty much what would be defined as the abstract function

Markus Sabadello: I think function that Justin has defined, function that Dave’s suggesting, I think I would be not too excited calling those two functions by different names, those are both resolution.
… Different layer of same function, one is abstract, other is more concrete… I think I agree with Dave’s point, feels inconsistent when it comes to metadata, input options, not saying anything about byte streams, what concrete function should be, property key value pairs, not saying they’re a byte stream whereas the DID Document does mention concrete byte stream.
… I remember from previous discussions that Justin had a reason to do that, can you elaborate Justin?

Dave Longley: defining the abstract function would quite literally involve copying and pasting Justin’s function and removing “byte stream” language – so i think the concrete function should instead just be layering type information

Dave Longley: on top

Dave Longley: (concrete type information)

Justin Richer: I think that means I should talk about Metadata structures, going to share my screen.
… If you look at PR #299, you will see all of the layers that I have… what I wanted to talk about was the fact that a metadata structure is at the same level of abstract or concrete depending on what direction you want to argue it from as string.
… Dave, please stay off the queue to tell me about a string.
… The idea is that functionally speaking, it’s a hashmap… strings for keys, strings for values. The reason for this, is that it’s implementable everywhere that I can think of. We’re not bound by specific encoding rules. We should say that that string is a UTF-8 string, define string, WebIDL does a good job of doing that, it gives us a byte buffer, that was suggested in one of the PRs, I think it should be added in at some point.

Dave Longley: i won’t get on the queue to point out that a “string” is a different layer of abstraction from a “byte array” <– but that’s the whole point, we should keep at the same layer of abstraction for inputs and outputs.

Justin Richer: The metadata structure is meant to be a specific kind of thing, in the way a string is a specific kind of thing. If you look at PR #299, every property in metadata structure has single name, and a single value. The name of the property is a string, the property names have to be unique.
… Reading Dave’s points about strings, you’re completely missing it, so I’m going to ignore your point.
… Names have to map to specific values, you could represent it as names, CBOR Map, any programming language, you can represent it with a simple native data structure. Why would the property values be strings?
… That’s something Markus and I have talked about, the reason for this is to get away from any thought about metadata being tied strongly to specific representations of DID Documents.
… However you manage string, JSON objects throwing into JWT, into blockchain, you’ve got it in a database somewhere, you’ve got a MongoDB, throwing string in relational database table - data that should be consistent across different implementations, different types of representation.
… A lot of what the group is struggling with, not having a way to talk about what a metadata property is, and what it’s value can be, we can start defining metadata properties.
… The resolution time metadata property is defined as a string in ISO Date format, we can start saying what these are in a way that is very clearly interoperable… the JWT spec does a really good job of doing this, of taking something that is just string properties/values in JSON object, number values, and put additional constraints to say, “This value MUST be a URI”, string that is parseable as a URI.
… Fine for a property definition to say things like that, as it gets passed into different implementations, I can take what you want and interpret it, if you want to parse and store as double precision timestamp, fine do that.
… without this, resolution is just an idea, an idea that everyone can implement and nobody can interoperate with.

Michael Jones: +1 to Justin’s comments

Justin Richer: And this has been killing the discussion in this group.

Jonathan Holt: This is my point, my challenge with VC spec, I ran into this problem, this attribute could be a string, or an object, stagnated my contribution… same thing w/ DID Document, right now, spec is very loosely, proof could be anything, in order to conform *tstr is what it is.
… That’s hand-wavy, more solid data model that could be interpreted into different serializations, we’d be further along, be able to prove interop.

Manu Sporny: going up to the high level, does the group want to define metadata I think absolutely we have consensus that the group wants to be able to define metadata and have a section on metadata in the spec
… the thing we’re getting tripped up on is the same thing as previously, which is at what layers do you define things
… do we have a difference between data model and representation or do we put those things together
… the group wants to define metadata in a section and it wants to concretely talk about that
… specifically there are 2 sections in the spec, created and updated, that belongs in resolver metadata
… there’s a very clear need for this section
… this section will appear, talk about created and updated and other resolution specific metadata
… the other thing it will probably talk about is the byte representation of this metadata

Justin Richer: q

Manu Sporny: we have to say it’s a iso formatted date time (or whatever)
… but in order to get to something concrete is we’re going to have to say somewhere in the spec
… the spec does this today
… this thing must be a URI, has an abstract portion and a concrete representation, you can test against whether something is a URI in the test suite
… the things we all agree on is we’re going to have metadata in the spec, it’s going to talk about these things
… either in a very concrete way, or split it up and have an abstract layer and a concrete one on top. The concrete one will be used to write the tests
… I think that’s what everyone agrees on
… The thing that I think needs further discussion is whether these are property-name property-value pairs or whether we’re going to support more abstract types on the right hand side like arrays or maps
… i get justin’s point, valid, which is let’s go for lowest common denominator which is a map of string-strings
… the counter argument to that is the vast majority of programming models that people are using also support arrays
… so the lowest common denominator is it string-string or can we support arrays?

Justin Richer: arrays are not a lowest common denominator because are your arrays typed or not?

Manu Sporny: raises the question that this thing we’re expressing is a map in the first place, so can we support maps as well for metadata?

Dave Longley: justin, could you please say why the document meta data wasn’t described as a byte stream in the function in PR 296 – would you object to changing it to that in your function definition?

Manu Sporny: clearly it stops at a certain place
… we are going to have to draw a semi arbitrary line at some point and decide not to support certain data structures, like graphs

Justin Richer: dlongley: yes I would object to changing that. It’s at a different level than the doc itself.

Dave Longley: that’s the core disagreement.

Manu Sporny: I would like to hear from other folks in the group about their thoughts on any of that
… what’s the appropriate layering? anything other than string-string property-name property-value things as metadata? do you have a use case where having string-string only is going to be problematic?

Dave Longley: function definitions should be consistent

Markus Sabadello: Just and I have talked about metadata topic, it’s defined in relatively abstract way, function returns did document metadata, doesn’t say anything about representation, doesn’t say if it’s http headers, json map, part of did document, it just returns metadata which has this model/features/properties that Justin described…

Dave Longley: +1 to markus, that’s the problem.

Markus Sabadello: Why does the method return data in abstract form, but returns DID Dcument in concrete stream, that is inconsistency that Dave mentioned, wanted to hear what Justin has to say about that.

Jonathan Holt: I think it would be great if ew can flush this out before we get to test suite.

Justin Richer: Metadata needs to be separate from document, once again Dave, we’re talking about different functions, not a DID Document yet, parse it, validate it, all of that is perfectly fine coming back… reason metadata not have rich datatypes, represent it… call function in code, can process it.

Justin Richer: sorry, another meeting I’m running

Manu Sporny: next steps… but it might be hard without justin
… next step is chairs and editors will discuss how we may proceed on this, including another special topic call potentially or some sort of chair/editorial decision
… thanks all!