16:01:45 <RRSAgent> RRSAgent has joined #did-topic
16:01:45 <RRSAgent> logging to https://www.w3.org/2021/03/25-did-topic-irc
16:02:03 <brent> Meeting: Decentralized Identifier Special Topic Call
16:02:11 <brent> Chair: Brent Zundel
16:02:24 <brent> zakim, this is did
16:02:24 <Zakim> got it, brent
16:02:38 <shigeya> present+
16:02:57 <markus_sabadello> markus_sabadello has joined #did-topic
16:03:04 <markus_sabadello> present+
16:03:35 <manu> present+
16:03:47 <brent> rrsagent, draft minutes
16:03:47 <RRSAgent> I have made the request to generate https://www.w3.org/2021/03/25-did-topic-minutes.html brent
16:03:48 <manu> scribe: manu
16:03:54 <brent> rrsagent, make logs public
16:04:14 <cel> cel has joined #did-topic
16:04:27 <manu> brent: This might be a quick call.
16:04:42 <manu> Markus: We could cover some resolution and dereferencing tests if we have time to do that.
16:04:54 <manu> brent: I think that sounds like a good idea.
16:04:56 <manu> manu: +1
16:05:40 <manu> brent: Let's jump in.
16:06:00 <manu> Topic: Resolution and Dereferencing Tests
16:06:39 <manu> markus_sabadello: Let's me try to see if I understand the test suite correctly.
16:06:45 <manu> markus_sabadello: I've been looking at the PRs
16:06:59 <manu> markus_sabadello: There are these default.json and defaultSuiteConfig.json that are inputs and the tests are run against those.
16:07:08 <cel> present+
16:07:09 <manu> markus_sabadello: I don't understand why we have both files.
16:08:22 <manu> manu: One of them is for human readable tests, other is for npm run test.
16:08:29 <manu> markus_sabadello: One of them is in an array, the other one is not.
16:08:42 <manu> markus_sabadello: My understanding is this is the input data being teested, then there are JS files with test files.
16:09:03 <manu> markus_sabadello: I started to work on separate suites for did resolution and url dererefencing.
16:09:29 <manu> markus_sabadello: I'm not sure if that's the right thing to do -- I put did-resolution and did-url-dereferencing into their own suites... all works in progress.
16:09:51 <manu> markus_sabadello: I haven't started working on the test code yet, started to work a bit on what input data should be... for resolution and would like to share what I'm thinking.
16:10:20 <manu> markus_sabadello: Want to see if it makes sense for resolution -- I'm thinking there is an array called executions... resolve in did spec ... array, for each execution, there is input and output
16:11:11 <manu> markus_sabadello: There is an array for resolve and resolveRepresentations... one main difference, executing resolve returns a did document data model, whereas resolveRepresentation returns a DID Document stream... in one of the concrete representations.
16:11:36 <manu> markus_sabadello: With this input data, I can write code for all normative statements... I could test that input DID must match the ID property in the resulting DID Document and other things.
16:11:48 <manu> markus_sabadello: So, accept, content type, etc.
16:12:12 <manu> markus_sabadello: One additional thing -- trying to annotate and mark executions as returning as being expected to return certain errors.
16:12:47 <manu> markus_sabadello: I'm saying that the second execution in this array, which would be notFoundExecutions, input that is not found, therefore output has an error... I can see if test code sees if this is true.
16:13:06 <manu> markus_sabadello: I could write test that checks, is the value not found, does it contain an error, etc.
16:15:00 <manu> manu: All of this sounds good, structure seems sane, you're testing in a way that makes sense... I'd just worry that Orie had a different idea of how this might be done.
16:15:14 <manu> markus_sabadello: My concern is that suites in VC HTTP API is groups of functionality... might make sense to do that here as well.
16:15:49 <manu> markus_sabadello: The tests seem to operate on these defaultSuiteConfig, would have expected tests to operate on fixture files... maybe that's where input data will go.
16:15:50 <cel> q+
16:16:14 <manu> markus_sabadello: Maybe Orie was trying to reorganize this somehow... anyway, at an early stage trying to figure out input data and output data.
16:17:10 <manu> markus_sabadello: There will be some relation with other tests... Manu, you had a comment -- some testing functionality might be nested or reference each other. For example, DID Resolution says output of resolve must be possible to run that through one of the production algorithms to get to a conformant algorithm.
16:17:35 <manu> markus_sabadello: The abstract DID Document data model, might need to take that and somehow model a production step and then test if the result is a conformant did document in conformant representation.
16:17:40 <manu> q+
16:17:56 <brent> ack cel
16:17:59 <manu> cel: This looks good -- wonder if it's redundant when errors are already in the output.
16:18:20 <manu> markus_sabadello: The idea is that will be tested... that says what executions have what sort of errors, and the result checks to see if that's true.
16:18:26 <manu> markus_sabadello: That says if the error happened.
16:18:32 <brent> ack manu
16:18:37 <brent> scribe+
16:19:27 <brent> manu: the conversion stuff, two things: the consumption tests are really difficult, you will actually need a consumer.
16:19:41 <brent> ... perhaps two people can write those tests, there aren't too many
16:20:05 <brent> ... we almost need to make pseudo implementations, not too concerned about that.
16:20:22 <Orie_> Orie_ has joined #did-topic
16:20:42 <brent> ... just highlighting that. For example you have a DID Doc data model and call JSON stringify on it
16:20:58 <brent> ... that would be one way to not re-use code elsewhere.
16:21:32 <brent> ... we don't want the test to be part of a lerge rseries. You could throw JSON Schema at it, but then you test a lot of things in addition to the normative statement
16:21:40 <Orie_> Here are our tests for jsonld consumption... https://github.com/transmute-industries/did-core/blob/main/packages/did-ld-json/src/did-ld-json.test.ts#L21
16:21:53 <Orie_> We had to implement part of the ADM to test consumption
16:22:02 <brent> ... the other thing is that it has been fairly easy to write JSON and JSON-LD tests. We haven't written any CBOR
16:22:02 <markus_sabadello> q+
16:22:03 <Orie_> q+ to propose import for CBOR
16:22:33 <brent> ... the current suite would require you to encode CBOR in JSON. The CBOR stuff may be a bit challenging.
16:22:46 <brent> ack markus_sabadello
16:23:13 <manu> markus_sabadello: regarding the encoding... trying to come up with resolve representation function data format -- thought for plain JSON content type -- could just do a string.
16:23:35 <manu> markus_sabadello: JSON and JSON-LD are easy -- resolve representation returns JSON-LD... just stringified JSON... for cbor, I used hex encoding.
16:23:47 <manu> markus_sabadello: but I haven't written any tests for that -- quickly summarize for Orie
16:24:11 <Orie_> Imo the hex encoding for cbor looks correct to me
16:24:13 <manu> markus_sabadello: Had some questions for Orie... started to think about Resolution and Dereferencing -- one question, the way I did it was by creating a different suite -- not to put it into main DID spec.
16:24:38 <manu> markus_sabadello: How does default.json and defaultSuiteConfig.json work?
16:24:42 <Orie_> code splitting is good! easier to test independly
16:24:54 <brent> ack Orie_
16:24:54 <Zakim> Orie_, you wanted to propose import for CBOR
16:25:16 <manu> Orie: I think Markus, you've nailed the how to test encoded variants... string encoded JSON, string encoded JSON-LD, hex-encoded CBOR, that's great.
16:25:31 <manu> Orie: Tests that iterate over fixture look at content type, decode content type, then decode properties.
16:26:06 <manu> Orie: Your question regarding default suite config vs. default.json -- the test suite is still not super well organized in that regard... I think we're using JSON config files here where we should be using JS file that imports smaller files.
16:26:46 <manu> Orie: default.json and defaultSuiteConfig.json are only different in external web server... one is an array and one is an object... if you're asking hosted version of test suite to run on suite config -- you pass suites you want to run, vs. just run individual suite -- but I think that's confusing.
16:27:03 <manu> Orie: Most people are not running these tests in hosted mode -- each org could host suite on test server.
16:28:01 <manu> Orie: We should get rid of one of them
16:28:16 <manu> manu: +1 to removing defaultSuiteConfig.json
16:28:40 <manu> Orie: There is a feature here where you can host the test suite on a web server and that's what these are a test of.
16:29:08 <manu> markus_sabadello: Another question I had -- different suite, different input file format... default suite config for existing tests -- is not enough for me to test resolution and dereferencing.
16:29:41 <manu> markus_sabadello: it doesn't have input/ouput data, doesn't have serialized items, but I think I might have created some redundancy -- media types, mime types,
16:29:46 <manu> markus_sabadello: maybe this is temporary
16:30:00 <manu> Orie: I think you did the right thing -- you want tests to be narrowly focused... test everything in one file, all possible options.
16:30:20 <manu> Orie: First version of this test suite, tried to test everything w/ one config in one file... hard to maintain, not easy to extend.
16:30:33 <manu> Orie: I think your approach works, is the right direction.
16:31:06 <manu> markus_sabadello: One part of it that's challenging, if you're a DID Method implementer, then you're going to have to go around to a bunch of different files to configure everything correctly... or maybe they don't support DID Resolution or DID URL Dereferencing.
16:31:26 <manu> markus_sabadello: Ok, then thanks very much, I haven't looked at many tests yet... feel confident that this can be done, will continue with that over next couple of day.s
16:31:27 <manu> q++
16:31:34 <manu> ack +
16:32:17 <brent> manu: a conforming producer and consumer language is har dto test.
16:32:30 <markus_sabadello> q+
16:32:44 <brent> ... one way is to focus on JSON-LD and assume that if there is an @context, they had to have it in their implementation
16:33:21 <brent> ... nearly impossible without code analysis of running code. This is pseudo=testable but got through. all I did was take a best effort at writing the test, then wait for others to disagree
16:33:35 <brent> ... maybe we can figure a way to test it more accurately
16:33:49 <brent> ack markus_sabadello
16:34:44 <manu> markus_sabadello: I would probably say it's not quite good enough, doesn't test what we're saying -- representation-specific entries, my assumption was test would model these different maps, would also need to be in the fixtures... map of properties, representation specific entries, as a result, here is JSON-LD specific... test checks given these two input maps, is result correct.
16:34:54 <manu> q+
16:35:20 <brent> ack manu
16:35:46 <brent> manu: we may end up discovering these things along the way. That is only testable in JSON-LD
16:36:00 <brent> ... that's probably okay, all we need are two independent implementations
16:36:11 <brent> ... I'll take your input and modify these tests
16:36:18 <brent> ... it is more accurate to do it that way
16:36:38 <Orie_> q+ to note about refactoring the test suite
16:36:48 <brent> ... I'm a bit concerned about how many of these things happen. Perhaps we will need (at some point ) to refactor the test suite
16:36:55 <brent> \ack Orie_
16:37:02 <brent> ack Orie_
16:37:02 <Zakim> Orie_, you wanted to note about refactoring the test suite
16:37:08 <manu> Orie: regarding refactoring the test suite, it'll happen -- many times
16:37:46 <manu> Orie: We'll need to increase convenience for Editors, test writers, etc... it'll get refactored along the lines of what PRs get into it. We're not saying any of the structure is set in stone.
16:38:05 <manu> Orie: As test report starts to cover more and more of the spec, people will become increasingly cranky of refactoring the test suite...
16:38:27 <manu> Orie: Rapid iteration on tests is critical, I expect we will change the way suite will work, inputs, etc.
16:38:38 <markus_sabadello> +1 to Orie
16:38:40 <brent> +1
16:38:47 <manu> Orie: Folks should be less concerned about "will this be ok forever" vs. "are we getting contributions at a pace that will yield results for us.
16:38:51 <manu> manu: +1 to what Orie just said.
16:39:00 <shigeya> +1 now understand the mode.
16:39:42 <brent> manu: could we spend a bit of time on understanding predicting what the long pole in the tent is going to be?
16:40:12 <brent> ... I'm looking at the tests, Orie, what are your plans for merging the spec tests in?
16:40:50 <brent> Orie: not planning on it
16:41:09 <brent> manu: Markus is progressing, I'm nearlyu done, Charles?
16:41:23 <brent> cel: I need to check in about that. I have a question
16:41:31 <manu> cel: There is some phrasing about ADM
16:41:46 <justin_r> present+
16:41:59 <brent> cel: some phrasing about the absrtact data model, a process needs to return something. I wonder how that should be represented in the JSON config
16:42:01 <manu> cel: That a process needs to return that... wonder if it's going to be ok to, how should it be represented in JSON config? Can we just use JSON representation, or special way to encode ADM?
16:42:06 <Orie_> q+ to answer how to represent the adm
16:42:16 <brent> ack Orie_
16:42:16 <Zakim> Orie_, you wanted to answer how to represent the adm
16:42:48 <manu> Orie: Our fixtures are in JSON and the good news is that JSON can represent the ADM -- the answer is "yes", if you need to produce an ADM, you need to represent it as JSON and be careful about how you communicate that...
16:43:10 <manu> Orie: Just make sure you're clear with name of variable... state is as didDocumentAdm or something like that.
16:43:19 <markus_sabadello> q+
16:43:24 <manu> Orie: So everyone is super clear as JSON -- we have to represent it as something.
16:43:25 <brent> ack markus_sabadello
16:44:00 <manu> markus_sabadello: The way I've done it so far is, when I'm dealing in ADM, I've been using JSON -- ADM can be expressed as JSON, when I'm trying to express concrete representation, I'm doing it as concrete reprenstation.
16:44:08 <manu> ... (of that)
16:44:25 <Orie_> yes, what a markus is saying is also good.... i think its good to use strings for concrete representations as much as possible
16:44:40 <Orie_> just be clear about what you are representing.
16:44:43 <manu> markus_sabadello: But sometimes, tests have to test JSON and JSON-LD representation, in test suite... we use JSON to express ADM sometimes, and sometimes we express it as JSON-LD (as JSON) -- we just want to be explicit w/ naming of variables.
16:44:45 <manu> cel: Ok, great.
16:46:02 <Orie_> we get to drop features that are not tested right?
16:46:11 <brent> manu: I'm a bit concerned about Daniel Buchner. They aren't difficult, but they need to get done.
16:46:21 <brent> ... sounds like we may have these in in a week or so
16:46:34 <brent> ... the only other thing is folks getting their DID methods in as well
16:46:50 <brent> Digital Bazaar plans to do a couple
16:47:06 <brent> ... it may be a good idea to have the test suite re-factored before then
16:47:26 <brent> ... order of operation is get suites in, then refactor, then get implementations in
16:47:39 <Orie_> 1. get close to 100% converage....2. refactor test suite..... 3. add did methods
16:47:40 <brent> ... I think we can be done by the end of April
16:47:46 <brent> ... in theory two more weeks
16:47:52 <brent> ... thoughts?
16:47:52 <Orie_> q+
16:47:56 <brent> ack Orie_
16:48:12 <manu> Orie: Yep, generally +1 -- we should get coverage reporting as close to 100% as we can before trying to do a big refactor.
16:48:25 <manu> Orie: Make a new suite that's very focus, like Markus did.
16:48:38 <manu> Orie: DOn't expect me to be the one that does the test suite refactoring.
16:48:45 <manu> Orie: Then we can add DID Methods after that.
16:48:47 <manu> q+
16:48:52 <brent> ack manu
16:49:14 <brent> manu: all that sounds good. There is one part I would like to refactor that will help folks submit their DID Methods
16:49:22 <brent> ... not really more than that
16:49:33 <brent> ... anything else people want to see done?
16:49:46 <Orie_> q+
16:49:54 <brent> ack Orie_
16:50:07 <manu> Orie: Inevitably, people won't write tests for features.
16:50:55 <brent> manu: one caveat, if Dan Buchner doesn't write the JSON I will have to
16:51:11 <brent> ... I don't know if Jonathan is going to write CBOR
16:51:14 <cel> relative DID URLs?
16:51:17 <brent> ... I thiink we have full coverage
16:51:28 <brent> Orie: should be able to look at a report
16:51:41 <brent> manu: not sure how to do that. we'll need to go line by line through the spec
16:51:50 <Orie_> here is an example of such a report
16:51:51 <Orie_> https://transmute-industries.github.io/did-core/
16:51:57 <Orie_> we should have something like this
16:52:03 <brent> ... for example, MArkus caugfht that I hadn't updated the normative sections for production
16:52:03 <Orie_> or declare failure.
16:52:09 <brent> ... people are contributing
16:52:20 <brent> ... are you expecting something will be dropped?
16:52:21 <markus_sabadello> q+
16:53:06 <brent> Orie: there needs to be a report so we know what's not tested. I wqas just saying the report won't be automatic
16:53:09 <brent> ack markus_sabadello
16:53:32 <manu> markus_sabadello: I'm not guaranteeing I can write normative tests for everything... lots of statements -- nextVersionId, last Update, I'll do my best, but we'll see
16:53:33 <manu> q+
16:53:34 <brent> s/I wqas just saying the report won't be automatic//
16:53:38 <Orie_> the test suite outputs JSON, it shows you if its tested... or not.
16:53:49 <brent> manu: there are a class of things that may be dropped
16:53:56 <Orie_> the report better be automatic
16:53:58 <Orie_> :)
16:54:03 <brent> ... canonicalID for example.
16:54:16 <brent> ... sup[er easy to cheat the test suite
16:54:26 <brent> ... I've seen it happen multiple times
16:54:45 <brent> ... folks can't point to an implementation
16:54:59 <brent> ... not sure how exacting the group will be on that
16:55:14 <brent> ... not sure how many people are lanning to implement equivalentID
16:55:38 <Orie_> We need to clearly communicate what is and is not tested... regarding statements and features.
16:55:39 <brent> ... that's the class of things that will end up being removed. Not becouse tests aren't written, but because implementations aren't there
16:56:16 <brent> manu: I don't know how to regenerate the output
16:56:24 <brent> Orie: should be in the readme
16:56:36 <brent> ... similar to other test suites
16:56:55 <brent> ... all automatic
16:57:03 <brent> manu: I couldn't get it working
16:57:12 <brent> ... I will read the readme
16:57:29 <manu> brent: Anything else for today?
16:57:34 <Orie_> https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server#usage
16:57:51 <Orie_> if the readme is not correct, PRs welcome to fix it :)
16:57:59 <brent> zakim, who is here?
16:57:59 <Zakim> Present: shigeya, markus_sabadello, manu, cel, justin_r
16:58:01 <Zakim> On IRC I see Orie_, cel, markus_sabadello, RRSAgent, Zakim, brent, justin_r, ivan, shigeya, dlongley, manu, ChristopherA, wayne, rhiaro
16:58:06 <manu> brent: Ok, that's a meeting, thanks all for coming - thanks everyone for your work on the test suite, you're all fantastic.
16:58:09 <manu> rrsagent, draft minutes
16:58:09 <RRSAgent> I have made the request to generate https://www.w3.org/2021/03/25-did-topic-minutes.html manu
16:58:10 <brent> present+ Orie_
16:58:27 <brent> present+ rhiaro
16:58:56 <brent> present+
16:58:58 <brent> zakim, who is here?
16:58:58 <Zakim> Present: shigeya, markus_sabadello, manu, cel, justin_r, Orie_, rhiaro, brent
16:59:00 <Zakim> On IRC I see Orie_, cel, markus_sabadello, RRSAgent, Zakim, brent, justin_r, ivan, shigeya, dlongley, manu, ChristopherA, wayne, rhiaro
16:59:25 <brent> zakim, end the meeting
16:59:25 <Zakim> As of this point the attendees have been shigeya, markus_sabadello, manu, cel, justin_r, Orie_, rhiaro, brent
16:59:27 <Zakim> RRSAgent, please draft minutes
16:59:27 <RRSAgent> I have made the request to generate https://www.w3.org/2021/03/25-did-topic-minutes.html Zakim
16:59:30 <Zakim> I am happy to have been of service, brent; please remember to excuse RRSAgent.  Goodbye
16:59:34 <Zakim> Zakim has left #did-topic
16:59:37 <brent> rrsagent, goodbye
16:59:37 <RRSAgent> I'm logging. I don't understand 'goodbye', brent.  Try /msg RRSAgent help
16:59:45 <brent> rrsagent, bye
16:59:45 <RRSAgent> I see no action items