Day 1 Notes

Open Annotation Community Group Meeting Agenda

Chicago, September 18 & 19, 2012

Agenda

Tuesday, September 18

• 09:00-09:20 Welcome & Logistics (Tim Cole)
• 09:20-09:30 Community Group Status (Paolo Ciccarese)
• 09:30-10:15 Style (Rob Sanderson)
• 10:15-10:30 Break
• 10:30-11:15 Context (Paolo Ciccarese)
• 11:15-12:00 Provenance (TBD)
• 12:00-13:00 Lunch (on own)
• 13:00-14:45 Multiple Bodies, Targets, Composite Annotations (Rob Sanderson)
• 14:45-15:00 Break
• 15:00-17:00 Multiple Bodies, Targets, Composite Annotations (Paolo Ciccarese)
• 17:00 End of Day 1

Wednesday, September 19

• 08:45-09:45 Semantic & Data Annotations (Herbert Van de Sompel)
• 09:45-10:45 Versioning (Paolo Ciccarese)
• 10:45-11:00 Break
• 11:00-12:00 Subclassing (TBD)
• 12:00-12:30 Serialization / Transport (TBD)
• 12:30-13:00 Wrap-up (Rob Sanderson, Paolo Ciccarese, Tim Cole)
• 13:00 End of Day 2, continue informal discussion if desired

Style

Rob presenting.

TODO: Link to slides

(...) - what styles are for.

Style: Currently applies to a target.

Issues:

• Makes specific resource harder to reuse.
• Style is annotation specific, so should be at Annotation level
• CSS Style resource is not valid CSS (no selector)

antoine: Think it should be on the target - but don't like either of the solutions! First a document with style information, then you annotate it.

Bob: The problem is that there are uses for both cases. "Make sure you look at 'this' part.". For the body - for making the distinction between a point in the body. Have to think about if it needs to be restricted.

Rob: OK, so we've been thinking about them for targets, but it could be applicable for both body and target.

Caveats: Style is explicitly only for visuailzations, not semantics

Bob: Then we have to face W3C accessibility Requirements, which we have not discussed/

Tim: One rational for attaching annotation, was to avoid having to say that the property attached to the resource might not be generally applicable to that resource. If I am targetting a web page, but I have reason to circle a red border when presenting that, I am adding that information in the context/purposes of my annotation. "This is the region I am talking about", make a new resource with identifer, to talk about that.

Tim: But when I am tryin to say something about.. CSS used for multiple purposes. "How to make it look for a user" but also how to select something. Is there a way to do both those things?

Jane: Not so much the style, but at the moment you got it attached to the Specific Resource. Can it also be attached to a target or body (non-specific).

Tim: Sometimes you want to attach it to the annotation. ,how to present this annotation - when you show this annotation, then show the target such and such. This is a different thing from talking about a target with a red line. -- so 3 different ways!

Rob: If attaching it directly to the target/body it would apply to all uses of that target/body - this is why we made the SpecificResource.

Rob: In order to reuse the specific resource that targets a selection of a picture, you would also get the style attached to it.

Anna: With standard CSS, we should not try to make people use some half broken version of CSS. What people expect is the open world assumption, so be careful about making style applying to every use of a resource.

Rob: Specic target is a resource described with a a state and selector, and in this proposal the style is done as :anno1 oa:styledBy :Style1 - for instance an oax:CssStyle saying:

@document url(SpTarget1) {
      color: red; 
}
@document url(Body1) {
      color: blue;
}

Tom: Like this, analoguous to how style is applied in HTML.

Rob: Not a normal structure applied with RDF - so you would have to retrieve the CSS to get the style. Not a common construct, but is part of CSS already.

Tom: For rendering the HTML.. you just paste whuch wiissch.

Rob: This is how I want you to render it - rather than the resource having it as a particular properties. Is this addressing the concerns of Antoine?

Antoine: Still a bit new to this, but it might depend on the documentation you make with this. If I want to annotate some HTML document, and I just (?) some text in the HTML itself, would I be expected to also make one of these?

Rob: Yes, if you want to render it in a particular color or style.

Antoine: What if the original HTML had red, then I don't need to do anything?

Tim: If you are annotating the entire HTML page, but you want to emphasize a part, you might want to do this with style, and not a selector. But if you are selecting a particular piece of text, you use a Selector. Then that provides a hint. (??). Trying to find a way to accomodate the requirement where style is more of a hint than requirement.

Antoine: The document is going a bit SHOULD over the place on this..

Rob: Probably should be lower-case "should"!.

Tom: What has confused people is the CSS selector vs CSS style.

Randall: Yes, are they different?

Paolo: When I used these in AO they were the same thing.

Randall: You might have a selector that is the CSS selector. (?)

Tim: CSS can be used for more than one thing. Even if it says Style it does more than style.

Simone: Do you have to do the style?

Rob: The application showing the annotaiton would have some defaults.

Simone: Could you say "Avoid".. A more high-levle style that is not this particular color, but avoid the state... (?)

Rob: Problem is when you are building an application which can't effect the rendering, say a browser plugin that overlays the image, it might not be possible to access say the color of the pixels underneath.

Paolo: So people in the RDF would not expect to find this.. you need to implement a parser for the CSS. The only example from 2004 I can think of is Fres Nell (?), Chris Peter(?) (NOTE: I think this is Chris Bizor; JGJ) came to MIT for a while, treating every single CSS as a rule, with a lens, that is applies. Honestly, this proposed solution is better, orthogonal to RDF.

Jane: Never liked having style and rendering hidden down in the model, like having this separate.

Bob: Having trouble in what is not included in the style. What is included is whatever is expressed. For example.. Is it a style issue if I want to annotate somethin that said that "Everything should have been in British English, but some is in American English" - and hoping that the consumer would do something about it.

Rob: That sounds more like a style request. A style would be more like "Render text in british english in green". This would not be "Please change this" - just when looking at the annotation, this is how to show it.

Tim: Yes, two interpretation. First case is to say that something is inconsistent; but the second is that it might be easier for my reader to highlight it such and such.

Bob: So it is all about Human Vision, which is a big problem.

Tim: Accessibility issue - so if someone is

Stian: CSS can also say things like "read in a male voice" or "read slowly" - but here that responsibility is moved down to the annotator.

Bob: And different systems might to do this differently.

Bob: Perhaps there are someone in the group already that knows about it; but not makeit something that cannot be solved.

Simone: imainging a situation where you have someone annotating a text and then wants to add a link to the body and wants to the link to be actionable, because the connection is part of the semantics, but when you have this mixed content situation, text + link, can the style help with this?

Paolo: can probably make use of the order [of instructions] in the CSS

Paolo: If i Have an (??) as a body, then .. I can have concurrency inbetween the two. If you write HTML you can have (??) - but there is an order on how to resolve it.

Anna: Most people use an XML serialization of RDF, why not a standard way to refer to CSS from RDF/XML.

Rob: Another issue is how to process the stylesheet, pllease render this XML using this styles

Stian: Just to be sure, you mean to apply the CSS to the RDF/XML directly?

Anna: No, but people might ask those questions.

Paolo: were does the style sit in the order of the things, when we render the body does it supercede?

Rob: So it would mean that the CSS in the Style1 would override the document style.

Simone: If the body is an HTML document,. it is used as the body of the annotation, if the styles you use... (?) - if you have a web page that is about a different page, and you want to link them, but highlight something on the page. The original stylesheet is good, but you want to use this as a body for a different targe tyou want to apply.. (?).

Tom: If we go with this, would defer to the CSS rules. Those rules can be quite complex with how to handle conflicts.

Paolo: So where do you inject it, before or after, makes a difference.

Tom: Yes, we should try this.

Herbert: Sounds bizarre to intersect - if someone has made the annotation, when it was seen it was rendered with some stylesheet, if you say now we are going to interject.

(??)

Anna: styles would apply to different media types in different ways, foir instance a mobile vs. desktop browser

Stian: Should how to apply the style not be given by us, but by the particular subclass of Style, say oax:OverlayCssStyle.

Tim: (?)

Paolo: If your resource exists, like a New York Times article, ..(?).

Tim: And usecases where there is no body. Acknowledge that it is an open issue.

---

Rob: Open questions are about naming and whether this should be in the core or extension. On the conference call oa:styledBy and os:hasStyle were mentioned. Any preference between these?

Anna:??

Bob: Not a big problem requiring something to be a class.

Tim: If we are going to have style as a class then should go with oa:hasStyle

Stian: would use oa:styledBy as it isn't the annotation that is styled bit the content deeper within it.

Jane: would put this in the extension

??

Paolo: Hope that the core will be more stable.

Rob: Core contains the base classes of the model. So will style be a base class? State, specifier, etc. are all web architecture and so are within the core spec. but style though, could go either way

Stian: About compliance as well - if *style is in the core, and you do style a different way, then you might not be compliant.

Anna:

(??):

Antoine: agree that there is pragmatic aspect, must convince users to use your stuff. Style, may look weird, complex to use

Stian: the more you put it into the core model, could turn people off. prefer simple core model that is simple for people to start using.

Rob: Proposed: oax:styledBy - oax:Styled, and one subclass oax:CssStyled - using the CSS document. AGREED.

DECISION: oax:styledBy - oax:Styled, and one subclass oax:CssStyled - using the CSS document

Context

(by Paolo)

Some annotation might have useful meaning only when the annotation target is considered in relation to the interpretative context. It might still be valid outside tha.t

Example: Paolo was on the football match yesterday. The picture heading has the wrong name for the player doing the touchdown. "This is Gronkowski's touchdown". Paolo can take down the picture and show it with the annotation elsewhere, and it is still good.

Say the picture annotation is "The touchdown was not Edelman's, but Gronkowski" - then you would wonder why I mention a different player.

Or annotating a picture on a profile page, "The image is wrong, as it does not depict Maryann".

Or what if the same picture is several times on the same page - and one of the images are included wrongly. "The image in the first column does not depict Dr. Maryann Martone".

Bob: This is why we want something about editors. What is wrong is about the label.

Paolo: Yes, you could sometimes rephrase the annotation - but for this editorial task it is ithe image that is wrong.

Rob: Yes, it is the image that is wrong here, not the text.

Antoine: (??) If you regard on this as an HTML page, then the annotation is on the <img> element, not on the image that is a source of that image.

Paolo: Yes, you could be precise or specific.

(??)

"The two images have been swapped" example.

Paolo: "This annotation is only valid in pages where the CNN thingie does not appear.

Paolo: An editor might just circle the image, and then send it back.

Sebastian: so if you talk about validity, what do you have to show the use as a validity criteria?

Herbert: The first example with the touchdown is a much better example.

Bob: How is this related to state?

Anna: There are other examples that are relevant to scholarly users. ??

Paolo: when I've worked in publishing, you attach the comment to the image with the text around.

Paolo: When reading this example, it does not make sense on its own.

Antoine: The annotator should not annotate just the picture - it should annotate the text that is wrong.

Simone: The annotator is looking at the entire page. The specific target is tthe picture.

Paolo: The annotator is not us.

Simone: So you are annotating the page. OK, this part of the page is wrong.

Tim: Highlight the text and annotate that.

Paolo: Yes, or both, these two are not c

Jane: At the moment do we support nested specific targets? "This section of th eimage as included on this web page"

Rob: Not forbidden, but not shown.

Randall: There is a composite selector

Tim: want to tell the processor to keep the image with it

Antoine: raising the use cases where the annotation tool is not doing a good job or the annotator is annotating the wrong thing

Antoine: if the annotator has the target wrong then they have the context wrong

Paolo: trying to support any type of annotation, cannot predict what the user will annotate

Stian: So let me get this clear, is your idea that the application puts in the context more like a kind of provenance, like "I don't know if this is important, but this is what the annotator was looking at"

Paolo: up to application, not that elaborate, what can I get out of it automatically

Randall: wondering if there is a need for the data model to change, because it sounds like the issues are all usability of the software, the context of where the annotator is annotating on the page are all expressable, except perhaps nested selector is not expressable.

Anna: There are usecases of annotating in a context, say a figure in a document. Which paper are you looking at.

Randall: Discoverability argument.

Rob: Say a PDF, not an image element - it is not about the section of the paper, but about the image that is a part of the doucument.

Anna: And within the context of the findings of that paper.

Simone: another kind of ambiguity

????

Paolo: unless you have a flag, or way to say

Herbert: is interesting to look as Antoine suggested from the user and application perspective.

stian: thinking that the context can be arbitrary, because annotating a specific wiki page in the context of the entire wiki, not certain why this can't be expressed using the current structure, e.g., the target is the webpage and the picture,

Paolo: target of the page becomes the target of the annotation and that is not what I meant as an annotator

??

Tim: so in a use case, I'm annotating an image in a paper and I want to reference the pdf of the paper without making it a target.

Stian: OK, so the context is a kind of optional thing to help interpretation.

Paolo: from the wiki, Jim has suggested xx:validOnlyWith, Jim's example nice because can scale with the use case

Jane: don't like the term "validOnlyWith", because the example was valid in any context

Paolo: only consensus with something with..

Bob: context is within the discipline of Psychology of...different readers have different methods of understanding the relational context; so two different annotators will have different context rules

Paolo: a little high [level] was hoping to accomplish something very simple

Bob: want to not exclude the possibility of flying high

Herbert: want to go back to Antoine's perspective; say you have a pdf file and you want to annotate an image in the article so that you have the uri for the pdf file as the target and a selector for the image; now say we have the same article in html, we have a very different way to express the target, very strange to have such a different way to express an equivalent thing

Simone: just contingent on the technology, from the user perspective

Paolo: right, but it is a problem in the model to have 2 ways to express the same thing in the mdoel

Antoine: Agree with Paolo: there should not be a different treatment for an HTML doc and a PDF one, just because the HTML makes the source image a bit easier to access (PDFs could make it easy too, one day...)

Sebastian: the difficult thing is when there are two different type of media, so for html the target is the image [see also Jim's notes in the chat] but in the pdf the image is embedded in the resource

Jane: ?? ...don't want to introduce ambiguities in the model

Tim: could use nesting of resources to do that

Rob: general agreement that nesting was possible but general opinion that the image is what is being annotated so if searching for annotations with the image as target then they should be retrieved, ..., prefer single nested approach because it allows the same sorts of things to be done with containing resources, as say a zip file, and so can annotate an image in a page in a zip file and so have a long chain of ...(resources?)

Simone: 3 different things going on:

1.  ??
2. user perception of what he is doing,
3. what we want to achieve with the model vis a vis the selector/target

Paolo: ...want to add in validity across a time period, e.g., annotation only valid as an event

Anna: agree, validity is important, a property of the annotation,

Antoine: isn't validity supposed to be handled by state

Paolo: state was originally called setUp and was meant to return to the state of the resource when the annotator annotated it; but here validity is different than what is expressed by state

Antoine: but it comes from a validity requirement

Rob: can use different selectors to ...?

Paolo: annotation might have a different life cycle

Herbert: did mention during the conference call that this could be part of state

Rob: specific resource that identifies some circular bit of something, which in turn is another specific resource, which is an image which is found with an xpath selector on a webpage, the problem is that the uri for the image doesn't appear (as the next resource down the chain is the hasSource)

Herbert: but could use the uri of the image as the identifier for the specific resource node

Rob: but if you do that then you are exerting the xpath selector globally

??

Paolo: this is also related to the fragments issue

Antoine: imagine your selector is a region defined absolutely, cannot work on the jpeg because of scaling

Paolo: can use hasState to get around this

Tim: we do have an issue that is a little open, especially with regards to videos

Paolo: ??

...??...

Tim: one thing discussed, say we have a jpeg at a certain scale, if we know how the user has scaled the image when annotating then can include that information with the selector

Simone: in terms of what is expressed in the model, having an image embedded in an html page, so that the image is embedded in an html is not a property of the image

Paolo: in general sense of context, open to state, but have the additional issue of validity, e.g., annotation is only valid in XX case

Herbert: can use state

Rob: but could end up needing states of states

Paolo: possible that the solution here could be applied to fragment...

Tim: let's take a break and come back to this

Rob: 2 proposals at the moment are here: general case, annotation of a cicular part of a jpeg in a webpage

1. first idea, specific resource selects the cicle in the jpeg and has a context/validity with the webpage
2. second idea, have a nested set of specific resources, so the specific resource for the the circle has a source and is selected via svg, the jpeg, which is a specific resource, selected from the html with an xpath, and in turn has source the html

...

Question that remains how to separate the case where the context is important from going down through the layers.

Tim: when we first added specific target it added a layer to find the thing being annotated; the second solution adds further layers

Bob: is that a problem for users or developers

Tim: developers

Jane: don't we need a third option because the first one is really just hasState?

Anna: can't write a sparql query for the second case in order to get the target back

Antoine: if you are a developer, model makes some assumptions about the hasTarget, but in the first option, it is confusing what is the target of the annotation

Tim: this is why we are discussing context, we want to know that we are annotating the jpeg, and note that it is in the context of the html

Paolo: see the point that the developer is working with the html but also need to query the annotation to get the target in a timely manner

Antoine: point is that it is unclear the option 1 is simpler than option 2

Stian: what is more important, the deepest element or the overarching element? The first approach, you can stop when you are satisfied, if all you need is the jpeg then stop there but if need the html then can encompass that too

Jane: so the html page doesn't need to be a specific resource, can be hasContext or something

Tim: so we should think about this across the afternoon and dinner and come back tomorrow

Rob: so we should have some people work on this. anna can you work out the sparql queries example

Anna: yeah

Rob: and so we need some volunteers to work on the other option

...

Herbert: In both cases have the problem of multiple places to look for the target

Anna:??

Herbert: already have oa:equivalent in the current model and so can use that to know all of annotations

Tom: one thing that is complicating this is the purist rdf approach; with a sql database can denormalize the database when building your queries, can do this with rdf too

Randall: question if you will reify all of the triples or just issue sparql queires against other peoples repositories, seems like the latter would be more desirable

...

Bob: ?? domain terminology

Provenance

Paolo: have asked ?? to provide some examples of mappings to the prov ontology; question that leyla and others had is where are the mappings, pointed them to the mailing list

• oa:annotator subPropertyOf prov:wasAttributedTo
• oa:annotated subPropertyOf prov:generatedAtTime
• oa:generator subPropertyOf prov:wasAttributedTo
• oa:generated subSubpropertyOf prov:generatedatTime

Antoine: provenance model is reinventing many wheels, would consider something more general, mapping to dublin core properties

Paolo: in the past have had files containing mappings but the problem is that the more mappings you have,..., there are going to be other mappings to other ontologies, need to decide where to put them in the model; prov maybe reinventing many wheels but is supposed to be replacing those wheels; dublin core was not enough

Tim: had similar issues, ... Nancy Ide to work on mappings to the Linguistics annotation model

Bob: ... some idea why the properties are not equivalent would be helpful

Herbert: ...

Paolo: always found dublin core lacking

Rob: twitter is the example I always use, see the twitter user handle and machine agent separately

Antoine: think that is the same idea as replacing oa properties with other ones; ... linked data group, when creating their vocabulary, used either new properties or existing ones, can use this method as a way to express differences

Stian: so in Prov, ...

??

Paolo: should we think of a general approach to provide mappings to things, should we provide a place for them

Tim: if the desire is to provide a human readable resource then can use...?

Jane: if looking at bigger context of something like generated, in prov very different from how oa is using generated

Paolo: yes cannot use them very loosely

Simone: onotlogy alignment, whenever you don't have an upper level thing informing what you are talking about is very difficult

Stian: so in prov, this one property is acting as shorthand for a set of processes

Herbert: we could truly use Prov's event model and would build on it

Tim: high risk for what we want to do

Herbert: yeah

Tim: have a use case for ...

Simone: should allow extension points, remain agnostic at certain levels so that things from other ontologies can be attached to meet your needs

Tim: yes, and should be attached in such a way as 90% of implementations can ignore them

...

Herbert: in this particular case, am wondering if we should be doing this [mapping to prov], comes with some baggage as jane was saying

??

Jane: the generated one is problematica

??

Stian: is in last call

Tim: generatedAtTime is an issue

...

Paolo: have one last thing, that is oa:equivalent

Stian: sounds like it is not black and white and this should be separate issue

Herbert: can you make a drawing?

Rob: different issue from provenance is what to do with equivalence; the case we use it in is anno1 is published to a server which generates anno2 then you have two annotations that are essentially the same but are generated at 2 different times (one by the client and one by the server)

Simone: there is a similarity onotology

Rob:

Paolo: stian, isn't there something in prov for this

Stian: yes, AlternativeOf

Rob: skos exactMatch and closeMatch are available

Antoine: not sure this is right, understood that this was to represent equivalence between resolvable and nonresolvable uris; but in this case is being used for resources made at different times; was wondering if ...

Paolo: were worried about people using sameAs in situations when sameAs was being used but the annotations are not the sameAs

Antoine: ...hidden issue is SameAs equivalent to oa:equivalent?

Rob: problem can cascade when annotations are harvested and instantiated in another server

Bob: scenario is very common; triple store with a sparql end point there is another...

Rob: don't want to require an end point

??

Simone: what are the contraints on sameAs

Rob: want to substitute the uris in the graph;

Herbert: another way to look at this is as an event driven model; rather than express equivalence, look at the process

Simone: but that means you must create the boundaries for "this"

Herbert: no, an event model, not actually advocating this, just saying this is possible

Rob: would want to go to an event model for other provenance issues too if we did this

Anna: if we are uising oa:equivelant for 2 different things for different times created and different formats

...

Paolo: should we have a mapping between oa:equivalent and prov:sameAs

Rob: don't want to invent our own but rather reuse someone elses

...

Stian: AlternateOf to say they are somewhat interchangeable

Anna: have to be careful to be specific about the what applies and doesn't apply

Paolo: yes, have to have boundaries about what is included

Rob: so thoughts on the possibility of using prov:AlternateOf, skos:

Antoine: so for things that are equivalent, they always have the same target and the same body, constraints...

Simone: must be clear about what are the essential properties and the conditional(?) relational properties

Antoine: ??

Paolo: in AO always trying to define everything strictly and here...

Rob: so does the oa:equivalent relationship imply anno2 have relationships to body1 and target1?

Antoine: may want to just express it in words, in EDM expressed several things as axioms; can be outside of the form

Simone: even the body and target are serialized in a different way, maybe they are the same but the structurally would be expressed in a different way

Herbert: think that the property from provenance is a good one in this case

Stian: what I'm hearing is that we have the annotation and ...annotating's outcome is an annotation, way it is modeled, serialization...has a generator (software that made it), attributed to some generator, and has an activity, the event of generation but also have an entity which is also generated, ... separating the annotation as an entity from its serialization, makes it easier to separate annotator from the software agent that generates the serialization

Simone: could this be allowed as an extension of the model; that is the important part, in some cases distinguishing between annotator and generator will be important but not in all

Rob: so don't need oa:generatedAt can be replaced by prov:wasGeneratedAt and then provide oa:annotator and oa:generator which are subPropertyOf prov:wasAttributedTo, new property for oa:annotatedAt, which is an implication of the prov:Entity chain

Stian: prov:wasGeneratedAt has range of a time; if you want to express a geolocation then need to include a generation event and so gets more complicated

Paolo: think of including geolocation (probably as a properties chain). (Rob: NOTE THIS FOR THE FUTURE)

Rob: so should break for lunch.

Multiplicity

Types:

1. Alternatives
2. Individuals
3. Composite

Have use cases for all three.

Multiple Bodies:

1. Alternatives could be translations of the body
2. Individuals could a text comment and tags that each apply equally to the target
3. composite could be a review and up-vote tag

Multiple targets:

1. Alternatives could be multiple copies of the same content at diff URIs
2. Individuals could be a single tag associated with many images
3. Composite could be a comparison and contrasting of multiple parts of a video

Selectors, states, and styles all have the same issues. e.g., alternative selectors are different ways that express the same segment, composite selector expresses that all of the selectors are needed

Multilpe Selectors:

1. Alternatives: different ways of expressing the same segment
2. Composite: All selectors needed (e.g., in video: SVG area + time)

Multiple States:

1. Alternatives: multiple archived copies
2. Composite: HTTP Headers plus timestamp

Multiple Styles:

1. Alternatives: styles for different media/device
2. Composite: All styles applied

Rob: Current model expresses these things very differently. Multiple bodies currently not allowed...except for the semanticTag hack, so in essence their is an exception to the rule; <throws up hands> no semantics for multiple targets at all. was discussed at length at the oac project review meeting but no headway was actually made.

Tim: an argument was advanced that maybe for many communities the distinction between these will not be significant. The distinction between car1 is red and car2 is red and these cars are red may not be significant.

Jane: we have a number of case studies in this area, have found these are neither individuals or sets; for instance comparing illustrations on greek vases, actually doing comparative analyses, similar in scholarly editions, might want have an order and annotating the relationships between these [this use case for juxtaposing resources for comparison in Emblematica too]

Rob: currently lumped under composite

Jane: but more complex than that

Simone: suggest to have a general relation between the targets which is left underspecified

Rob: hold of on the solutions discussion for the moment while we first clarify the use cases

Anna: we store all the relationships between the targets within the body using an RDF body

Rob: other parts of the model where we have multiplicity, selectors, we explicitly understand that this is a choice and have a composite selector for selectors to be interpreted as a set.

Jane: what if they have an order

Tim: can be tricky to express order

Rob: so we also have style where we do the opposite of selector and multiples are understood to require all of the styles as a set. and so we have a mess

Anna: in css there is an order to the style, and we don't say anything about ordering the multiple styles

Rob: would be nice if the semantics were clear, we need to have modeling for choice of resources, individual resources, sets of resources, and a graph of resources; a big hole has been left in the model and it needs to be filled

Tim: what is the difference between targets and bodies with respect to alternatives and individuals; all of the copies could simply be separate annotations

Paolo: if I'm creating an annotation, and I'm pointing at three targets, the point was to keep track of it being a part of a single annotation event

Tim: For alternatives, just show one, each translation is applies to the target equally, so these are processing instructions, how far do we want to go with processing instructions in the model

Simone: so what is being annotated vis a vis multiple targets, if annotating a set, then annotate the set, same for the graph example, with indviduals, have to express the relationship between the individuals, are the alternatives, or...?

Antoine: an annotation has a target but not negating one of the statements if multiples are choices; is not actually possible under rdf

Rob: any graph implies all sub-graphs

Anna: not true because you aren't telling it which to use,

Simone: more a problem for the client to pick the right one, not sure if the semantic can communicated to the client, just pick one

Tim: if the annotation applies, e.g., if selector1 applies to specific target1 is ok, if selector2 applies to specific target2 is ok,

Simone: can we have a real world example

Rob: if selector1 is a media fragment selector and selector2 is an svg selector for the same region

Paolo: problem, how do you understand if the annotator has a preference for which to use, for instance i have an application that makes a polygon but it is smart enough to know that not everyone can consume the polygon...

Antoine: two ways to select the same thing, but isn't it just one selector with two ways of being represented

Rob: would need the abstract notion of the rectangular selector and that you have a choice; in the composite case the selectors are different selectors

Jane: do we need to have it? we have the property oa:equivalent, do we need a oa:alternative? or oa:preceeds?

Rob: with preceeds then you have the ore:proxy problem; you can end up with an infinite nest of selectors;

Jane: so why not alternative?

Rob: so you can have the case

Anna: are we going down the rabbit hole, the number of use cases for multiple selectors is finite; maybe if it becomes a use case for a large number of people, can reconsider then

Rob: that is fine for selector but need to consider the model as a whole [how to cope with multiplicity not handled consistently across the model]; so what about a composite target?

Tim: not good because what if I have 10 pictures of frogs and I want to +1 them

Simone: so why not just have 10 annotations

Rob: but then you have alot of annotations, and what if you want to express them as a set, then you need to add them to another annotation

Antoine: semanticTag seems to change the semantics of bodies, even without the need for multiple bodies, so the semanticTag should be the only body

Paolo: yes, that is where we are going

Rob: the issues for the composite node solution -- the user can't decide what is meant, client has to check more structures when consuming, doesn't degrade gracefully, distinction between individuals and alternatives needs more classes, allows arbitrarily complex tree structures, not backwards compatible

Rob: consider an offset type of model, adding a node that denotes AND; still allows for arbitrarily complex trees but does degrade gracefully and is backwards compatible; if we require all of the targets/bodies then don't need the links from AND to the targets, degrades gracefully; can do this for selectors, etc.

Rob: so only need to mint 5 new properties:

1. bodyComposition
2. targetComposition
3. selectorComposition
4. stateComposition
5. styleComposition

and new resources

• oa:Alternatives / oa:Choice / oa:XOR (only use one of the resources)
• oa:Individuals / oa:Bag / oa:OR (all of the resources apply individually)
• oa:Composite / oa:Set / oa:AND (all of the resources apply as a set)

On the pro side, model is not significantly more complex, consistant for all classes, allows explicit declaration, allows especially multiple bodies withour losing semantics; on the con side not common pattern to have a predicate modify interpretation of the graph structure

Bob: extremely common in biology,

Anna: given that this is so much about the processing in the client and not the truthiness of the

...??...

Antoine: don't see the need for three options, can do away with Individuals

Rob: another con, no identifier for the compisite itself

Stian: if in the core model, you apply them then you need to understand them

Rob: need to have the default values spelled out in the core model, then have the new properties and resources in the extension document

Herbert: at least then we would have an understanding of what we mean

Paolo: take html5, not everything is implemented but everything is in the spec

Stian: what to do if you get a structure you don't understand

Paolo: you ignore it, the advantage of having this node in this place is that you can ignore it if you don't understand it but if you have a node in between...

Simone: something weird about how the annotation is being expressed not really saying anything about how the things in the annotation are related

Rob: right so version2 isDerivedFrom version1 so what is the target? you cannot target the relationship, the fallback is version1 and version2 are the targets

Stian: the thing that i like is that the body is something about the targets

Bob: have more of a worry, where would the formal semantics go if the about was formalized, can really see the problems with multiple multiples, other thing, cannot make a distinction between OR and AND, led to the more general question, when there are users that have to use it with formal reasoning, will this be six different owl representations? don't see that we are saying that we would like to have a standard OWL representation, would be happy to see a natural OWL DL representation, and even simple rdfs reasoning may run afoul

Rob: to have a predicate in the graph that affects the nature of the graph, seems to be breaking what rdf does

Tim: or breaking new ground

Stian: exactly how to interpret the relationships seems less important than having methods to find them

Bob: no order to the nodes and arcs, the real magic is that it is more of a style, as anna was saying more an instruction for processing

Herbert: so as an interchange mode, this is more of a shortcut, makes the transportation of the graph smoother and easier, what is shared can vary from how the internal storage and processing works, just communicating some specific processing instructions; so this is not about how the data is to be stored but how to be shared

Bob: call it sometimes the international diplomacy model of annotation; there are contradictions at the domain level, for instance the way the model worked, then whether or not you have and and or you end up at or

Rob: the difference for me, is for or, if you miss one of the ORs it doesn't matter but if you miss one of the resources for AND then your processing stops dead

Tim: the offset graph doesn't really affect how the graph is interpreted but the more succint version does

Rob: link to annotation is to show that the AND applies to that annotation

Anna: another problem with it though is that the AND is then modifying the targets for all other annotations

Simone: what was the main problem with the composite target?

Rob: becomes arbitrarily complex; when consuming the annotation need to determine if the target is a target, specific target, or something that might contain either of those, the sparql query become very difficult

Simone: could insert the AND between the target

Rob: essentially like the AND solution

Tom: can we look at the combinations of these

Jane: concerned about the additional complexity

Herbert: can ignore it because there will be defaults

...??...

Paolo: Two things, would want some validation of the second model is very different, and also if I ignore the targetComposition then I might misintrepret the annotation whereas the other simply makes me stop

Antoine: that is a feature

Paolo: scared to misinterpretation and the last two proposals are very susceptible to misinterpretation

Anna: isn't that a feature though?

Paolo: just saying it is more prone to it

Rob: seems like a profile issue

Herbert: two ways to look at this, one someone is going to consume it into a repository, and the other has implications for how to render it

Simone: even with the composite one it is just a big arc so that it isn't such a big issue

Paolo: can give counter examples , when you marshall graphs into projects, ...?

Rob: Shared Canvas has an example of the composite model, sc:ContentAnnotation, the body is a choice

Tim: has been pointed out that the first proposal, because some of the relationships can be ignored by conusmers, easily gets us to the composite model so would be a better option

Simone: using the composite model, in order to avoid the complexity, ... seems more natural that the difference is clear, while the semantics of the first proposal are unclear

Tim: I would claim that the first proposal covers the composition case but includes the current model better

Antoine: the beauty of it is that it allows choice of where to use hasTarget, lets the publisher control completely what is consumed

Simone: so living underspecified with hasTarget, doesn't say anything about the relation about those targets but set makes a relation between some of them

Tim: don't have a compelling use case to talk about the set

Rob: would argue that proposal 2 doesn't specify things for communities but allows for the naive case; the point is that any resource can be annotated, if you can understand the resource then you can process it; can explicit in the naive case and not preclude the naive case

Simone: in the core model all that is needed is the target

Antoine: in oax would have the composition with two examples AND and OR

Herbert: but then couldn't put it the extension document because the multiple targets is core

Antoine: so the mutltiple is interpreted as individuals and composite handles AND and OR

Tim: that has been discussed in the past, builds in more indirection for the sparql queries

Antoine: pattern for composition is a way to hide that behavior

Paolo: have a simple model, annotation with multiple targets, is like foaf:members, no construct there, just a list of members, the individuals case, no addtional semantics specified, but if we need to say that the targets are a set then add a node that says that the targets are a set, can do the same for the OR case. this is more rdf compliant, but we can allow the mixed case in the event that the publisher doesn't care how the consumer interprets the targets. on the downside, the producer can nest structures

Bob: so it allows arbitrary boolean functions

Rob: sounded fine until you allowed the hasTarget relationships that target multiple composition nodes; how does this work when applied to selectors, states, and styles

Tim: the good thing about it is, if you have a video, then you may have a situation where you have a composite selector and then an alternative mediaFragment selector; need to be cognizant that what is done for selectors, states, and style should be consistent with what is done for bodies and targets.

Bob: [has counter example, would also apply to 'choose your own adventure'-style narratives]

Simone: wanted to stay compliant with the model while adding semantics for compositionality, using a sub-property of hasTarget that connects another target which connects to both targets, it is the same as Paolo's example but it cannot force a decision;

Stian: so like a specializedTarget; only allows one level of target, individuals by default

Simone: so if you understand specializedTarget then use that

Paolo: but if I query then I get both and don't know how to interpret

Tim: could be dealt with by breaking it out as its own property; so do we want to say in the core multiplicity is individuals but do we want to say the specializations are in the extension; new predicate will need to be in the core

Paolo: good with that but is still a target, but calling it differently because it is more comfortable

Bob: is there a reason to limit the level of [nesting for] targets? worried about excluding taxonomies

Some consensus emerging around simone's example: xor and and in the extension document, individuals in core; will re-examine tomorrow

action items: proponents of examples 7 and 19 should look at the implications for specified resources and when the body and target are related (e.g., when body and target are specified resources of the same resource)

Linguistic Annotations

strong history of annotating text, audio, and video; power users of annotation tools, LREC 2012 (1/6th of papers were about annotations); thousands of implementations of specialized annotation tools

tools of linguistic annotation

• praat
• elan
• field linguist's toolbox

natural language processing

annotation being used to analyze speech

annotation density measured, interval marking, manual microannotation of speech

natural language processing annotations

• word annotations (tokens)
• span annotations (markables, e.g., phrase)
• tree-like annotations
• relational annotations

each of these would be a domain set of annotations

oa may be too verbose for linguisitics area

challenge in linguisitics - conceptual interoperabilty

COAT - CoreOnto Annotation Toolkit (developed in Korea), for semantic annotation

difficult to find incentives for linguistic annotators to take up oa

some possible reasons they should

• reuse implementations
• adjudication workflows
• automated calculation of inter-annotator agreement
• quality control / error finding in NLP tool chains
• annotation provenance
• conceptual interoperability

current linguisitics efforts

• NERD (Named Entity Recognition...)
• Lemon (Lexical dictionaries)
• OliA
• Apache Stanbol
• ITS 2.0 Ontology

current anno frameworks

• NIF (NLP Interchange Format)
• POWLA
• LAF/GrAF

very difficult to extract metainformation for NIF but

• easy to implement
• provides best practices
• extensible

but

• unsuitable for adjudication of annotators

Default values in OA will be useful to minimize the lossy nature of mappings into othe standards like NIF

Day 2 Notes

Multiplicity

option 1

explicit uuid for set / choice; default settings but no fallback

option 2

fallback position, uuid for choice, set, list, or individuals

option 3

targetInterpretation property, doesn't allow for nesting though

Anna: for option1, it is impossible to use sparql to query for all of the targets

Paolo: want to not recommend to go farther than that, if people need to nest they will

use other structures, if they need it they will find another way; can recommend only 1

level of depth

Tim: Can leave individuals in the core and push set [union actually] and choice into

the extension.

Herbert: This makes alot of sense for body & target, but doesn't make sense for state,

selector, etc. then should the choice/set be in the core.

Tim: don't want selectors and targets to be in conflict, but don't have to have all of

the options

Herbert: does anyone like 3?

Paolo: nice that it is compact but very unnatural

Herbert: if we don't like then we should get rid of

[option 3 out]

Tim: should we discourage non-annotation uses by saying that people 'should not' make

deeper layers of bodies and targets

Stian: may have the opposite effect when you tell someone they cannot

Bob: question about option1, how do I write a validator to decide that I am compliant

with this model; how to tell what is a composite and an individual, a composite has a

typed target?

Rob: the vast majority will be a single selector or a choice

Tim: do we have a use case of individuals for selectors, maybe we should disallow them

Stian: if individuals are meaningless for selectors then can say they are unspecified

Jane: do we want more consistency in the terms

Rob: was already decided to use set [not composite, although union or composite more

expressive...]

[continuing multiplicity of selectors discussion]

Rob: does discontinuity make a difference?

Tim: an important use case, applies a rationale for why we have to modify selectors

Rob: Implication is that a list could be [unordered]; can we write off option2?

[option 2 out]

List option being added to option1.

Tom: if we are writing off ... are we better off embracing the rdf:container concepts,

because it seems like we are reinventing

Antoine: these things are being revisited in the rdf groups, could leave these

containers in and then see if rdf:containers can replace them later

Bob: the formal semantics of the rdf:containers are all the same, they are a human

reading ...

Rob: things to keep in mind are that we have a default value under option1, List could

be a sub-class of rdf:list;

Tim: what are the options for ordered list though? how to do that?

Antoine: would recommend postponing this, it isn't about putting the order out of

scope, just delay the practical methodology for applying it, can associate it with an

rdf list containing the order

Rob: punt on the technical part

Tim: but going to be asked how to do the discontinuous sentence, if this doesn't go

into the spec then at least need to provide an answer

Antoine: is good, can just go into the documentation

Stian: an oa:list is the same as the item but ...?

Tim: advantage is that the ordered part can be in another namespace

Context

Paolo: one solution is to procvide a specific target with prov:AlternativeOf; another

solution is an undefined property for html

Anna: problem with nesting

Tim: is the jpg the target or the html, one or the other will have the uri obfuscated

Tom: imagining that a user will want to find all of the annotations for the webpage;

argument is that they are annotating the larger thing

Rob: heard anna say we are doing nesting things so surely we would use specific

targets, list, etc. but no place to put the jpg in this example

Jane & Anna: like the top option

Randall: in that example would it be bad to allow more than one hasSource relationship

Rob: wouldn't know which source to use then

Paolo: undefined property in the example could be a specific target

Tim: could use occursIn

Rob: what about annotatedIn

Tim: annotatedOnlyIn would mean that the annotation is valid only in the context

resource

Anna: makes sense to have an occursIn relationship, perhaps if you are annotating an

image and don't care about context then you can use state

Tim: or provenance

Anna: there are use cases where you will want to annotate an embedded resource and you

need the contextualizing resource that it is embedded in

Antoine: so what is really important is the annotation that target's the jpeg;

Tim: moving into listserv with some variation of this oax:annotatedIn proposal (outward

selector)

Jane: validity being left as an open issue

Anna: think it will be a big deal

DECISIONS: oax:annotatedIn proposal - deferred to listserv; validity on backburner

until a critical mass builds to force it to be handled

Data & Semantic Annotations

Presented by Herbert -

many use cases for machine understandable bodies already have various ways to handle this; e.g. using hasSemanticTag another approach - body is a graph, has rdf:type trg:Graph structured body approach - blurs into triples that can no longer neatly delineate the

relationships

proposal - discontinue support for structured body because can express it using named graphs, etc.

Antoine: not ideal but should not be ruled out

Herbert: very different, talking about the content of the annotation, the problem is

that we don't want to use it to express metadata

Antoine: but there will be cases that need it

Paolo: should be clear that there could be problems, conflation, etc.

Bob: problem is in open world, need to continuously analyze the graphs because the

change over time, so not the best idea

Herbert: one proposal is to get rid of hasSemanticTag

- issue with non-information resources vs. information resources

Anna: symmetry on this problem for Targets

Rob: for example the target of annotation is the real Eiffel Tower

Tim: makes sense to have sub-properties for body and target that differentiate non-

information vs. information resources

Antoine: but why consume as an annotation?

Anna: have a use case for annotating the work

Antoine: but this is what rdf is for out-of-the-box

Jane: raises a more fundamental issue for our model, when to use and when not to? when

to record something as an annotation or as metadata?

Paolo: with a graph, you can do everything we are doing here, we do this because we

want a certain structure

Herbert: introducing hasConceptualBody (will be used for Semantic Tag) and

hasConceptualTarget

Anna: Tag is somewhat orthogonal for hasConceptualBody

agreement - tags are orthogonal

Jane: authentication gets quite complicated with threaded annotations; haven't

discussed much how nested annotations are affected by the proposals here

Paolo: need to experiment with this

sub-classing

Decision:

Motivation - sub-property of Annotation, cardinality 0 to many, no oax sub-classes of

oa:Annotation, all annotations must have rdf:type oa:Annotation

remaining issues

State (time/http working document)

serialization & transport (document on wiki)

Versioning (document on wiki)

Zooming/scaling ?? - Jane: specify additional information in state in order to specify the scale - Antoine: cannot be taken care of by the selector? - Anna: really a property of how the image is rendered - Tim: zoom level can be done on the server side or the client side, probably should

think of state as how to do http headers in state and zooming/scaling as whether or not

it should be in state - Antoine: seems to be overloading state - Anna or Jane to add their name to the zooming/scaling issues

specific resource identifier ??

accessability ??

fragment identifier sub-classing

can have many sub-classes but may not be able to keep up with the standards or can simply link to the page for the overall standard

going wqith option 2 - pointing to standard document (profile link), extending this to everything

planning

next phone call - topics 4, 8, 11

phone calls every two to three weeks