Meeting minutes
Implementation Reports
<ktk> w3c/
<gb> Issue 142 Implementation reports (by rubensworks) [enhancement] [needs discussion]
ora: AndyS you raised concerns about impl reports in last meeting?
AndyS: yes. we need impl reports. have we asked for them yet? do we wait for CR?
… we ought to have a way to direct implementors somewhere
pchampin: the publication as CR is a call for implementations.
… doesn't mean we couldn't be more explicit about that.
… having clear instructions on what we expect would be useful.
… not sure if rdf-tests repo contains those instructions.
… assuming we'll expect EARL reports.
… ideally we should document this better. document how to submit.
… raises question about what we want to do with the repo. separate the leaving CG test suite from the frozen WG test suite?
… there is pressure to be more strict about github orgs. if it's a CG repo, it should live in the w3c-cg organization.
… w3c should be reserved for WG stuff.
ora: is there guidance from w3c on what happens when CG conclude and a WG is constituted? changing the URL of the repo doesn't sound like a great idea.
pchampin: no such guidance.
… separation of orgs is relatively recent.
… the repo was primarily a WG repo, except when the WG existed. CG would be fallback to maintain it.
… CG chairs seem to have a different point of view.
… what is leaving test suite?
… so far we decided to work together on what used to be the living test suite.
… as WG we need to freeze it now.
… alternative would be to embrace rdf-tests being a CG repo. should move to w3c-cg. but now that we're moving to CR, we should make a snapshot of this in a WG repo.
ora: how do we do that? fork the CG repo? have our repo track that?
pchampin: I don't have a strong opinion.
AndyS: not advocating a way. but useful to get them sorted out before there's confusion later on.
AndyS: the repo goes back longer than this WG. it's been around a long time.
… it does have changes in SPARQL 1.1 (maybe also RDF 1.1) not from the last WG. continue to be evolved.
… if this WG takes a snapshot, make sure we know what we're snapshotting.
… could do everything.
… for implementors responding to call for impls, wasn't sure if we were calling for them to report on their 1.1 status, or just focus on 1.2 features.
… I don't think there's too much difference for many impls in the WG.
… thinking about teams further out. we were getting questions about 1.1 suite recently.
… there was a SHACL 1.0 impl report submitted in the last couple of weeks. there are times when people report quite a bit later.
… helpful for a group about semantics. we can crudely do it by taking git commits, but that is unfriendly.
… sounds like we need two repos. can't have a formal area inside CG repo.
rubensworks: about impl reports for older versions, I noticed recently in rdf-tests, we have lost the old impl reports.
… we had them in that repo, but with the restructuring, somehow they have been lost.
… not sure if we want to restore them at some point, or in anothe repo.
… regarding other point, makes sense to move the current repo to w3c-cg. make a fork of that repo here under w3c. maybe as a git tag.
pchampin: I'm not sure I said that we need a snapshot. I was reporting what I thought I heard in conversation.
… the RDF 1.1 test suite has evolved under the CG before this group was created. some tests fixed. some added.
… my understanding is we are curating the tests for the spec. the understanding of the spec is getting better, we are improving the test suite.
… any new RDF 1.1 impl should rather parse the latest version of the CG test suite, rather than a snapshot of the historical suite published 10 years ago.
… the test suites are not codified by the process. they are not frozen like a REC is.
… not claiming we need to freeze it. might be a good signal, especially as we are requesting impl reports. not nice to change tests people are supposed to run during CR period.
… apart from that, I'm happy to have a single repo which would be WG's repo. in absence of the WG, we rely on community.
… not pushing against either option, but really like both. both have merit.
… fork by WG gets me nervous. would mean it may evolve on its own. then CG might want to get back some of the changes made by WG. would want fork to happen when we decide not to make more changes.
ora: when people submit reports, they should make clear against which suite they ran. which version. which tag.
… if it's not a 1.2 suite, then impl report isn't useful for us at this point.
… I would say if we publish instructions on how to submit impl report, it should say "make sure impl report includes which suite you ran against"
… from w3c standpoint, where repo is is important. but not as important for this group.
… as long as we know what was run against
pchampin: we need to know which version of the test suite was used. could a date be enough?
… I appreciate that git history is not always linear. not trivial to map a date to a commit. but are we in a corner case where this would not be good enough?
ora: ultimately the question might be how much does W3C care?
… if a date is fine, I'm fine with that. and an indication that they ran against a 1.2 suite.
pchampin: for indication that they ran 1.2, usually in our reports is a report on each individual test. for each test, we need 2 impls.
… as soon as each test is sufficiently covered, we know we are good.
… not which test suite, just which tests.
… as long as each test has 2+ impls
ora: so we could have 3 impls, none complete?
pchampin: yes
ora: need instructions on how implementors can do this.
… is the idea to fork the repo, put your report in a directory, and create a PR?
rubensworks: yes. making a PR is what most people do. in the past, we also accepted reports by email.
… PR and adding a report at a specific location in repo should be fine.
AndyS: somebody is going to not use github. in some ways respecting that decision means we might need anotherr routine.
ora: so email as a backup?
AndyS: if we agree, we can wing it some way. encourage PRs.
pchampin: that's the point I wanted to make.
… would it be possible to ask people to provide it as a patch?
… people who don't use github don't have anything against git (?)
… slightly easier on us.
AndyS: might be easier sending it as a file rather than a patch.
… doesn't have their name on it, but neither would a patch.
ora: put instructions with details.
<pchampin> +1
<niklasl> +1 for allowing git-patch mails too
ora: now we need somebody to write the instructions. volunteers?
pchampin: good example could be found. looking at json-ld...
… we had these instructions previously.
rubensworks: if not urgent, I can look at it when I'm back from holidays.
<pchampin> instructions for JSON-LD: https://
ora: I will try to take a stab at it sooner, but if not you can do it when you get back.
pchampin: gkellogg had something doing earl to html. we can try to reuse his ruby code.
… or something different if maintaining ruby is going to make it more complicated.
AndyS: at one time there was a script to do it. cannot find it now.
TAG's advice on error processing
<ktk> https://
ora: brief discussion on this.
pchampin: I sent an email to Yves L. and to mailing list. Gathered some good arguments to show that what is good for CSS isn't necessarily good for RDF.
… trying to port "graceful degradation" to RDF is not necessarily a good idea.
… that being said, I believe we could try to work on option 2 (discussed last week). not "this is the graph corresponding to invalid data", but when we see an error, return data from a controlled vocab about strategy parser applied.
… I gave 3 examples. idea would be that those terms would be IRIs we define. other IRIs would be allowed.
… we can stick to our position that graph produced by invalid syntax is undefined by spec, but we show TAG we tried to do something about improving interop in a way that is more inline with RDF.
ora: so define IRIs and allow it to be an open set.
gtw: one thing that is not clear to me: at what level does this remark from the TAG play out?
… it seems to me that these IRIs would come as return values of API calls
… this would be hard to specify beyond providing the IRIs and advise people to use them
ora: that's what I imagined.
pchampin: totally agree. by discussing this with others, I came to conclusion that at runtime, this info would rarely be used.
… could imagine that those IRIs would be referred to by documentation of a parser library.
… imo, that would be OK. it would be inline with how we respond to the TAG.
pchampin: about testability, not all tests need to be automatable. we could also ask in the EARL reports some manually generated reports.
… could be codified in EARL that the test was run manually.
rubensworks: wondering if option 1 is still on the table? option to just fix it in the spec, define a specific behavior.
… I understand the reasoning for option 2, but I'm wondering if there's really a big need for consuming applications to have such behavior.
… will it actually be used?
… might be more complicated than we think. behavior to continue parsing after an error might have subtleties that would be hard to specify.
gtw: to pchampin's point; the part of this that would be easily testable is cases where the behaviour is not to continue
… there are some proposals on github which would be incredibly hard to test, and too complex
… In my mind it would be a very small vocabulary of IRIs.
… It could be used in the SPARQL protocol, but only for IRIs that indicate a stop.
pchampin: to rubensworks point, option 2 is complex is precisely the reason I'd reject option 1.
… some applications will not find what they need. that's a good reason to keep this vocab extensible and be very flexible in what parsers can do.
… they need to tell us what they do, but can do a number of different things.
pchampin: about how useful this is, I'm not sure it's very useful at runtime, but while writing the app, if parser library behavior is documented (recovering on n-triples errors)...
… if what I'm doing is open-world, behavior is OK for me. If I'm doing SHACL, a partial graph is as bad as an empty graph.
… potentially I'd just drop whatever the parser gives me.
… I do find this useful to document. I hope current parsing libraries document it to some extend. We'd be providing a vocabulary to describe it.
pchampin: to gtw, the case where the parser is continuing is for me a good example of what we call a 'warning'. I can imagine several classes of signaling the error.
… some cases would be errors, some warnings. Just a different way to talking about it.
… just because the parser continues doesn't mean it looks like a success.
pfps: pchampin described seems to be what we have right now.
… if you signal an error, stop.
… how can we do any better than that?
… should we do better than that? if there's an error in input, it could be for all sorts of reasons.
… specifying what happens if you emit a warning seems fraught with corner cases.
… for n-triples not bad, but turtle? what do you do if you see a malformed prefix? unclear what you should do.
… if you see malformed piece of a large block, what do you do?
… all sorts of corner cases.
… I'm in favor of 'let's do what we've done.' If you see non-compliant data, signal an error to consumer or you signal a warning and consumer is free to do whatever it wants with what you produce.
… I don't see anything more specific than that that is going to be useful.
ora: the way I had thought about continuing after error, signalling is at the end.
… I'm worried about distinguishing between warning and error.
… important that the client knows that not everything went well.
… if something is signaled, something didn't go quite right. IRI would tell you what you got.
pchampin: I agree option 2 is not adding much to what we have.
… trying to come up with fine-grained classification of behaviors is not something we want to do.
… how useful would 3 classes I proposed be, I'm not sure.
… I agree we don't want to be too specific.
… it's a tradeoff in terms of being more specific, and what could be acceptable.
gtw: ora, you said "it's important that the client knows that something went wrong"
… there are so many kinds of APIs;
ora: if you're suppressing errors, and something comes back from parser, no error was signalled.
gtw: if I asked my API to ignore all errors, why should the API be forced to include the signal with the triples it returned
<pfps> Right now, it seems to me that we have three classes - valid input, error signalled, warning signalled
pfps, no, we don't have a error/warning distinction
… I don't think the spec should be concerned with this kind of API design considerations
lisp: I would be uncomfortable with document that proposed concrete IRIs for things like this
… if the document wants to say 'if the process completes, indication that it completed'. progress (100%) or 'up to a certain character', that is implicit in the language definition.
<pfps> pchampin, The spec says - A processor which detects errors on input MUST signal them as errors or warnings.
lisp: notion something is being returned is not there. to go to extent that we have identifiers for things we don't know what they are would leave me uncomfortable.
ora: not in the API design business here.
lisp: if you want to go and say that, the GSP is a protocol. In that document that would have a place for it.
ora: earlier we said we're basically entertaining adding this as advice to implementors rather than something mandated.
… if this is OK with TAG, I'm all for it. if parser encounters an error, when signalling the error, should provide information about how processing was done.
lisp: if authors wanted to provide discursive descriptions of failure modes, that would have place in document.
… not IRIs.
… that would provide implementors/users with things they could look for in their APIs to respond.
ora: these IRIs are shorthand for predefined descriptions. open-ended set.
lisp: different state of mind as to specificity of the definition.
ora: still strongly in favor of option 2.
… understand discursive description comment, but think we're not necessarily excluding something like that.
… I look at it as just shorthand.
<ora> STRAWPOLL: Error handling: Option 1 (as discussed earlier), Option 2 (as described by pchampin), Option 3 (something else, incl. "discursive descriptions" as suggested by lisp)
<ora> 2
<lisp> 3
<pfps> Option 1 - no change to spec
<tl> 2
<rubensworks> 1 (but could live with 2)
<AndyS> 0 - I don't know the implications - e.g. does the TAG expect existing parsers need to change? do we need to have tests?
2 (but could live with 3)
<doerthe> 0
<ktk> 1
2 (though I'll admit to feeling out of touch with the specifics)
<Tpt> 3 but can live with 2
ora: good starting point for discussing next week.
regrets for next week
ha. thanks ktk. lots of writing today on a small laptop keyboard! :D
<ktk> no worries