DID WG Topic Call on Issue Processing Working Session — Minutes

Date: 2021-02-11

Attendees

Present: Markus Sabadello, Ivan Herman, Brent Zundel, Shigeya Suzuki, Manu Sporny, Amy Guy, Orie Steele, Dmitri Zagidulin, Joe Andrieu, Daniel Burnett, Justin Richer

Regrets:

Guests:

Chair: Brent Zundel

Scribe(s): Amy Guy

1. PR progress

Brent Zundel: today we’re looking at PRs that close out issues. Jump on the q to discuss

Manu Sporny: I have updated all of the issues, I have some questions
… there are 5 horizontal review issues, not touching those yet
… my expectation chairs and staff is that we are going to cover them at the tail end of the editorial cycle
… people will make one final pass to make sure they’re good, correct?

Brent Zundel: brief update, as far as I am aware horizontal review opportunities have been offered to all of these groups
… PING met to discuss us earlier this month
… they have yet to publish minutes about that so I don’t know what they said
… but looking at their slack they haven’t mentioned raising issues
… if they had something they wanted to say they would have said it by now
… the other horizontal review has been completed

Ivan Herman: the main reason we kept these open is because we will have to give a report on those when we make the request for CR transition
… we will have to make a reference to each of these because each say they cleared horizontal review
… they only reason they were kept open is administrative

Manu Sporny: deferring to the chairs and staff on what they want to do..

Ivan Herman: close as soon as chairs and me finalise the CR request

Manu Sporny: sounds great
… I have also marked a number of issues as defer-v2
… we let everyone know we were going to do this
… want to make sure we didn’t mark anything that was being actively worked on, that someone got a PR in last minute that addresses one of these

Joe Andrieu: there is one for privacy considerations for service endpoints
… there is a PR in

Manu Sporny: anything else on this list? herd privacy?

Joe Andrieu: I commented

Manu Sporny: jonathan for the DIIPS? no on the call
… ACLU responses.. joe and adrian?

Joe Andrieu: that’s appropriate.. it’s a very subtle point and we’ve tried to address it in some of the use case stuff, nothing that has boiled up to spec, deferring is appropriate

Manu Sporny: when should DID params be dropped, Orie?

Orie Steele: I think this has sort of been solved by the pulls to the key rotation section and examples
… we could safely close it
… generally query params in DID URLs are a poorly supported and understood feature and anyone building on this is playing with fire

Manu Sporny: good to know, we’ll keep it open and defer, because there isn’t a definitive yes we dealt with that
… and bring it up for a new WG
… this public key id type duplication is defer v2, that’s mike jones
… A couple i did not defer
… because they are in process
… shigeya is updating the architecture diagrams, that PR is waiting
… that’ll be in there, it’s editorial
… the deterministic canonical cbor hasn’t been merged yet? that’s strange, I think it was, just missed this one
… proving control sections is an editorial thing that needs to be fixed, can be done in a week or two
… PII is correct, editorial
… IANA considerations there’s a PR in
… wasn’t merged because of merge conflicts
… things like this are okay, they’re not disruptive to what the editors are doing, they improve the spec
… so I wasn’t going to defer this one unless people objected
… those are all the issues, we’re down to like 18
… many of them are addressed. Only deferring like 5 issues, which is pretty amazing
… Questions for the group, are you okay with the state of things? Are issues being processed per expectation?

Brent Zundel: I’m very pleased. 618 is just a continuation of what’s already happening
… not a new thing
… and it’s editorial
… bug fixes

Manu Sporny: agree
… On to the PRs
… The massive editing cycle has begun
… there are 2 PRs I want to talk about in a bit
… I have started marking all of the PRs as editorial, substantive or non-substantive
… this is not something we really need to do, but it’s a training exercise for the group
… anything marked as substantive, if we do that when we’re in the CR process it is almost certainly grounds for doing another CR
… it’s bad
… any time you see this label after we go into CR it’s not a good thing
… Want to show what those types of PRs look like vs the non substantive ones
… which can be normative, but don’t change implementations
… eg. this non-substantive change removes duplicated normative requirements
… we state you must do x, and later on we say you must do x
… same statement repeated twice in two places
… we remove one, implementations wouldn’t have to change
… A substantive change is we made normative changes and everyone’s implementation is going to have to change as a result
… In one of these, we had no normative reqs around DID URL syntax
… we said what it is but didn’t enforce it
… so added a normative statement that says this is the syntax you musts conform to
… everything that’s editorial is meant to be editorial - feel free to disagree
… the editors can miss stuff pretty easily with mass cleanups, accidents happen
… or a change where the editor in their mind has a good argument for why it’s editorial but it may not be clear from the PR
… this is the editors attempts at classifying the PRs and highlighting which ones people should pay mor eattention to vs not
… questions?
… everyone happy with this process for the next 3 weeks?

Brent Zundel: comments or questions about the process?
… From my perspective we’re following the same notions that we followed all along. We want to ge tthings doen as reasonably quicly and efficiently as we can, with minimal process
… which is why our primary process directive is that if you see something in that shouldn’t have gone in, let us know
… it rarely will be too late
… with that in mind, we need to keep up with these things
… those who have opinions need to state those opinions. Subscribe to the repo and you’ll get email notifications

Manu Sporny: timing of these PRs
… we had said earlier this week during the main call that we’d like a 24 hour turnaround but it may take longer
… the editorial pass is happening top to bottom in the spec
… that means the top part of the spec has a whole bunch of PRs in on editorial cleanup and changes
… I’m tagging people that probably have comments on each one of those sections and everyone is doing a great job engaging
… thanks for reviews, they really help
… these PRs are going section by section
… it’s ideally only three to eight paragraphs of text in each PR
… a lot of it is formatting, line lengths
… but some do rewording
… if you’re tagged please do the review
… I’m trying very hard to not merge things in that have not had at least two other parties do a review
… the whole 24 hour thing, some have been floating for 2 days now
… some is that the editors don’t need it merged right away so we’re giving a bit more time
… as a general rule I’m trying not to merge things that don’t have multiple reviews
… anything that is a substantive change will probably be left out there for at least 48 hours
… especially if it has people objecting
… any objecting across any PR is not merged
… all that to say, the rule I’m working with is if at least two different people have reivewed the PR in addtion to the editor of the PR and everything looks good, no objections, and changes adopted, I’ll merge it in
… that usually takes around 48 hours, based on data we have so far
… any concerns, questions around timing around PRs?
… we can just delve into PRs that people are working on

2. production and consumption issue

See github pull request #596.

Manu Sporny: There are 2 that have raised the question around ADM again
… markus, would you like to provide a background on both of these PRs and the current concern?

Markus Sabadello: one of these PRs, production and consumption
… trying to show how production and consumption works and what the relationship is between the data model and the individual representations
… the data model defined in section 4 and the representations in section 6
… the data model we define as an infra map
… not json, not cbor, not xml, it’s infra notation
… the diagram is attempting to show how this abstract data model by going through a production process gets to a concrete representation in json-ld, json and cbor
… that’s it, the discussion and controversial aspect is around whether @context is included in the ADM
… if ADM has to be present in the top left corner in order to produce the json-ld representation
… that’s the discussion

Orie Steele: its not about production… its about consumption… the consumption arrow is where the error is.

Markus Sabadello: I think the consensus after the ADM discussions was the data model, the infra map, can hold @context, or other representation specific entries, but it doesn’t have to be there
… this diagram is trying to show the simple case where the data model contains core properties and other representation specific entries they get added in individual representations

Orie Steele: i’m one of the people who doesn’t think that this is a simple diagram
… the consumption arrow is the problem
… the production arrow is fine
… how methods choose to represent context internally before production occurs, is up to them
… context is a representation specific concept
… consumption rules we have spent time talking about preservation of properties an dthe fact that multiple representations means that unregistered properties are all preserved
… the arrow violates the WG consensus
… it sounds like @context must be present in the ADM to produce json-ld. I’m not saying that
… this picture implies somehow you can consume json-ld and lose an @context
… we’ve admitted we’re going to conserve all properties
… I’ll object until we can fix that visualization
… on markus’ original point, @context doesn’t need to be in the ADM for production
… but if you’re consuming json that contains a member into the ADM and somehow losing it, that is a problem

Manu Sporny: +1 what orie said
… it’s a difficult balance
… what we’re doing here is putting 3 different production/consumption things together

Justin Richer: for what it’s worth, I am in favor of markus_sabadello ‘s diagrams and do not agree with Orie ‘s changes.

Manu Sporny: because the diagram is trying to show production and consumption across multiple representations we should at least put @context in here because in this way, even in consumption in this way, you’d have @context
… the change request is to put @context in the ADM
… it makes it possible for the reader to ask the question why is @context disappearing in production here
… and not here

Orie Steele: I would not put @context in the ADM… its confusing… I would split up the diagram

Manu Sporny: it could also be that they raise the question fo wait a second you just consumed something where did this @context come from?
… I think there are down sides both ways
… but the way it is currently proposed makes it seems like @context appears out of thin air
… we can’t propose that as how this thing works in general

Markus Sabadello: I made another version of this diagram
… a subset of the other diagram that doesn’t show all the production and consumption at the same time
… but shows what happens when you consume the json-ld, and @context is preserved, and you produce a plain json representation and the @context is still there
… this is the agreement we had after the discussion
… I still think it’s a mistake, but that’s what we agreed

Orie Steele: A series of diagrams seems more helpful

Markus Sabadello: I think it’s more important to show a diagram that starts with a data model that does not contain any representation specific entries
… if we put a diagram like this in a prominent space, people would ask why we have something represent specific in an ADM

Ivan Herman: +1 to markus_sabadello

Markus Sabadello: we could have two diagrams
… one that looks kind of like the one in the PR and in addition something like this
… that shows preservation of the entries

Brent Zundel: in any simple diagram of a complex system you’re going to be making generalizations and making things less precise
… I think either we clarify with the second diagram that says in this specific case this is what the spec text says
… or add a sentence that says this diagram is a simplified visualisation and pay attention to spec for specifics

Manu Sporny: +1 to what brent said
… and +1 to this diagram, I like it much better
… having a series of diagrams will be easier for people to not get confused over
… I was going to say in this diagram I’d be fine also with the second thing eliminating @context
… I know that’s controversial and some people would disagree
… it is an issue with round tripping
… if all of us can look at this diagram and say we’re fine with that being in the spec I think we’re done

Orie Steele: I think a series of diagrams would help
… we need the other picture. This picture works for json-ld production and consumption
… deleting @context will work for json production and consumption. Maybe there’s a CBOR version to show as well
… It’d be better to tackle the complex case of what happens if you start with json and you want to produce json-ld in a diagram that’s just about that
… as opposed to a diagram with 6 arrows and hiding what’s going on
… that’s confusing
… as far as manu’s proposal for allowing you to produce did+json that drops a property, that violates the consensus of the WG
… but I’m in favour of a picture of this that’s just vanilla json that never has an @context in it ever

Ivan Herman: I am sorry but I don’t like that one
… if I am coming from the outside what I see is that the application did+json and did+ld+json are exactly the same
… if I know a little bit about json-ld then I don’t know what the @context does and plays what role on the right hand side
… If I remember well, we separated the representation specific properties form what is in the core set
… what about making some sort of a diagram which has the data model as it is now
… without the context
… and in that box in the upper left, a separate entry somehow for representation specific properties
… where the context is present
… we do separate these things
… context is specifically called out as a special thing in the spec

Shigeya Suzuki: Isn’t it possible to keep representation-specific property as part of didDocumentMetadata ? (I mean, split the upper-left box)…

Shigeya Suzuki: ah, that’s what ivan is saying I guess…

Ivan Herman: in the representation on the data model there is something else that should be part of the diagram
… when I produce the json I use it or don’t use it
… and things become clearer
… but merging the two things for me is a mistake

Justin Richer: +1 to ivan
… the underlying problem people seem to be having with the original diagram is that the json-ld crowd sees this as okay so obviously you’re starting with the thing in the lower left and everything else is derivative and you’re dropping properties
… that’s not an accurate reading of the diagram
… which means the diagram is misleading if people can read it that way
… what this is trying to show is that there is a DID doc which is the thing in the upper left

Orie Steele: I agree with Justin… the diagram is confusing.

Justin Richer: that can go in and out of representations using the production and consumption rules specific to those representations
… I like ivan’s idea of putting the representation specific properties into a separate bucket in the ADM
… we had long discussions about these being different things
… than did doc properties
… I think that would help to tell that story

Orie Steele: I also agree with Ivan and Justin’s proposal to store @context in the ADM in a separate bucket.

Justin Richer: because I completely agree that the other one that has @context in both the json-ld and json isn’t telling an accurate story either
… it is telling a limited story of limited usefulness to make one small point
… about carrying this @context value through
… if you’re producing did+json then @context is a meaningless field, which will be held next to the data model, not in the data model
… and to go in the opposite direction to produce ld+json from a json source, you would need to create the context probably based on that field that was passed in
… but potentially not
… because there will be plenty of cases where the plain json has no semantic labelling in it
… and that needs to be added by the ld processor on production
… I get where the LD community is coming from, but that’s not the story I believe markus was trying to tell with the original diagram
… which is why I am against the proposed changes

Markus Sabadello: I can update the PR with two diagrams based on that
… what’s most important for me is to make the point that you can start with the instance of the ADM that does not contain any representation specific entries and then produce all of the conformant representations from it

Justin Richer: +1 to get rid of the consume arrows

Markus Sabadello: maybe orie and manu would be okay if this diagram did not contain the consumption arrows
… or it would be greyed out and this would show that based on the ADM you can produce all of these
… and as a second diagram we could have representation specific entries that are preserved
… I can also do what ivan said, to illustrate that this @context is part of the data model but is representation specific entry
… I could do that if it’s acceptable to have two diagrams
… the second PR that’s open is about the same thing
… all the concerns we heard apply equally to the other PR
… which is another diagram more to do with resolution
… but same underlying issues, I could also split that into two diagrams

Brent Zundel: what if we added a description to the diagram that says: “This diagram illustrates how the abstract data model might be used to produce each representation. For specific details on the production and rules for each representation, please refer to …”

Orie Steele: Splitting consumption and production up would help a lot
… in the production case, going from a box that has context floating next to the ADM to a json representation that does not contain a context, that’s very valid and exactly what we agreed to in the WG
… if we remove the consumption arrows we’ll be on the way to solving issues
… we already put @context in its own special box
… it’s a property that is a representation specific property allowed in the ADM
… some of the language using was using is challenging because we only have one bucket to put ADM entries, so we’re shoving representation specifc entries and DID doc entries into the same bucket
… at this point we’re stuck on that path
… but it doesn’t mean we can’t have a nice clean picture of production and consumption that is respective of the native representations
… I don’t agree that it’s an LD thing
… I think it’s a JSON preservation of properties in a consistent manner

Justin Richer: if it’s not an LD thing then just drop @context :)

Manu Sporny: if it’s not an LD thing then just drop foo :)

Manu Sporny: ^^ see the problem w/ that line of argumentation?

Orie Steele: if I add foo or $schema and do consumption then production, I expect those properties to preserved by JSON
… I care more about the json rules making sense
… json needs to preserve properties when being consumed and produced through the ADM

Manu Sporny: we’re getting to something maybe workable
… there are normative statements int he spec that are probably going to have to change
… I think it might be for the best
… our data model keeps thinking that, sort of asserts that the DID doc is the only thin you can express and that’s the root
… with the work on the resolution section and this stuff I think it’s clear that we want more than one bucket
… the data model is capable of expressing a number of things - metadata structure, DID doc, Verificaton method structure
… or a bucket for representation specific properties/entries
… as long as the group is okay with this direction, I can make some substantive changes that create multiple buckets
… make the ADM a bit more generalized
… and I think that will address a number of issues throughout the spec
… I think there’s something workable here
… markus if you rework the diagrams
… and I work on some of the text
… we may be able to come to something more generalized that can be applied more cleanly across the entire spec
… +1 for the changes requested
… and making the diagrams more specific

Orie Steele: selfissued is not here… but he would agree with me :)

Justin Richer: respond to the ‘foo’ property thing
… this is, as has been previously argued many times, as we’ve talked about, it really is a different kind of thing
… say your source doc is coming in in json and has a property named @context: true
… this is acceptable, it gets dropped into the ADM with a property named @context with a boolean value of true
… all of that works
… when you go to produce that into ld+json
… everything is legal so far
… when you go to produce ld+json that LD producer has to supply a context field and it has to have a valid value
… and it is programmatically going to have several sources of information for the value of that field
… extra properties dumped into the data model that the consumer didn’t know how to do anything special with, like @context, is going to be one of those
… the problem is that the LD produce absolutely has to understand the @context property if it shows up in the ADM
… and be able to process it with a correct value
… so this is not to say this is just about property preservation
… it’s not
… pretending that it is is not helpful to the conversation
… what we’re really talking about here is how do we create an LD json document
… when we have different sources of information for what goes in the context field
… which is semantically significant to json-ld, but not necessarily to anybody else
… all of these examples of what you want to happen are coming form places where you’re getting good data
… but you’re not guaranteed to get good data from other implementations, other representations
… this is a dangerous road to walk down

Manu Sporny: I disagree with Justin, but won’t belabor the point – the WG has discussed this many times, and has shown how @context is useful in JSON-only. We’re just not going to agree on this.

Orie Steele: I obviously don’t agree with justin_r, luckily we both appear to agree to the application/did+json production and consumption rules in the spec.

Manu Sporny: Yep, agree, let’s just leave it there.

Justin Richer: apparently we disagree on what the WG has agreed to as well

Markus Sabadello: I will try to work on updates to the diagram
… manu I was surprised to hear you think normative statements have to change
… I think we’re already saying this n the spec
… we say the data model can contain two kinds of entries
… I thought that was what the third diagram was bout, which is less controversial
… in an open PR that already shows the specification today, that the data model can hold the core properties and also representation specific entries
… I look forward to any suggestions on how to improve also the text
… I want to show the other diagram this is controversial but the same underlying topics

Justin Richer: +1, I don’t see any need to change normative statements from these diagrams

Markus Sabadello: the same thing as before, production and consumption fo different representations, but adds a resolution here
… resolve which returns ADM and resolveRepresentation which returns a representation
… probably the same issue here, which if you consume a json-ld the @context should be preserved
… so i made another version, one of the sequences is you resolveRepresentation you get back json-ld, consume that and produce json with @context preserved
… I can also try to update this PR with input that was given

Manu Sporny: +1 to that diagram, very helpful
… I’d be good with that
… The normative statements I’m talking about, we just want to be crystal clear about this discussion
… in section 6.1 we say:

Manu Sporny: We say this in 6.1 currently – “The representation MAY define representation-specific syntax that can be stored as entries in the data model. These entries are included when consuming or producing to aid in ensuring lossless conversion.”

Manu Sporny: when we say data model we have other language that says the data model is the DID doc

Orie Steele: “stored as entries in the data model” => DID Document : INFRA<MAP>

Ivan Herman: +1 to manu

Manu Sporny: we are not clear at all that there could be different buckets
… the clarification we need to make is what those different buckets are and potentially where the production rules pull from when they serialize and deserialize
… there are normative statements in the spec right now that lead to the discussion we’re having
… orie and I being surprised about the diagram
… we’ve had the discussion, seems like everyone is on the same page, let’s just make it explicit about how the group thought about it
… maybe not every single clarifications is a normative statement, but there might be a few
… I’ll put them in as a substantive edit and see if it helps us align more
… everyone can look at it

Brent Zundel: another PR?

3. Questions on Shigeya’s diagrams

See github pull request #612.

Manu Sporny: shigeya, thank you for putting those diagrams together, very helpful for the architecture

Brent Zundel: +1 big thanks to Shigeya for his PRs

Manu Sporny: one concern is that it’s a black and white diagram, colour coding may help
… the simplified diagram still has a number of arrows that look scary
… I know the point was providing relationships between everything
… can you provide the source other than svg? google doc? that would allow the editors to iterate?

Shigeya Suzuki: sure
… it’s in my branch of my repo
… already available
… if you want them in main, I can commit, or mail it to you

Manu Sporny: ideally the sources are in the repo.. what tool are you using?

Shigeya Suzuki: using OmniGraph on Mac
… it can export several formats, I will inform you which format is available

Manu Sporny: if it can get into google draw, that seems to be a useful thing
… if not we can recreate it
… we just want to get it into a format we can iterate on
… The other question was layout with the diagram, I’m sure you’ve tried to lay it out in a way that’s easier
… I’m wondering if we can simplify further and reduce arrows to the minimum
… it could be that DID URL and DID URL dereferencing could be a different diagram, down in the URL dereferencing section
… that may simplify the diagram further
… ideally I think we want 5 concepts in the diagram. 6 to 7 is a bit much
… at this point we have 15 including the arrows
… I really appreciate all the work in the diagram
… trying to see how aggressively simplified we can get

Brent Zundel: +1 to the idea of separate similar diagrams for each concept

Manu Sporny: so people look at the diagram and see it as simple on the surface

Brent Zundel: any other feedback on this?

Shigeya Suzuki: manu says you were going to edit the drawing for colours
… if there’s any colour scheme you want I can add before I provide you?

Manu Sporny: I don’t have any suggestions.. whatever looks pretty

Shigeya Suzuki: sure. I suggest some warmer colour scheme, my preference

Manu Sporny: thank you

Justin Richer: +1 to black and white

4. DID URL syntax issue

See github pull request #621.

Manu Sporny: request that people review the PRs, especially substantive

Brent Zundel: want to introduce one?

Manu Sporny: DID URL syntax.. cosmetic changes, putting things in the table
… the normative change that we didn’t have before is that all DID URLs must conform to the DID URL syntax ABNF rules
… if you look at thE PR it can be lost with all the formatting changes
… this is the normative change to look out for
… search for the word MUST and see if something has been added or deleted, and see if you agree
… I do try to summarize why it’s been marked as substantive in the PR

Brent Zundel: we’ll close, thanks all
… Please look closely for the next few weeks at our repos, a flurry of activity will be present
… we hope you can all remain engaged
… thanks all!