Streaming JSON-LD

W3C Working Group Note

This version:
https://www.w3.org/TR/2020/NOTE-json-ld11-streaming-20200507/
Latest published version:
https://www.w3.org/TR/json-ld11-streaming/
Latest editor's draft:
https://w3c.github.io/json-ld-streaming/
Test suite:
https://w3c.github.io/json-ld-streaming/tests/
Implementation report:
https://w3c.github.io/json-ld-streaming/reports/
Editor:
Ruben Taelman (Ghent University – imec)
Participate:
GitHub w3c/json-ld-streaming
File a bug
Commit history
Pull requests

Abstract

JSON-LD [JSON-LD11] offers a JSON-based serialization for Linked Data. One of the primary uses of JSON-LD is its ability to exchange RDF [RDF11-CONCEPTS] data across the Web. This can be done by first serializing RDF to JSON-LD, after which data consumers can deserialize JSON-LD to RDF.

Since RDF datasets may contain many triples, and JSON-LD documents don't have size limits, such documents could in some cases become very large. For these cases, the ability to serialize and deserialize JSON-LD in a streaming way offers many advantages. Streaming processing allows large documents to be parsed with only a limited amount of memory, and processed chunks can be emitted as soon as they are processed, as opposed to waiting until the whole dataset or document has been processed.

The recommended processing algorithms [JSON-LD11-API] do not work in a streaming manner, as these first load all required data in memory, after which this data can be processed. This note discusses the processing of JSON-LD in a streaming manner. Concretely, a set of guidelines is introduced for efficiently serializing and deserializing JSON-LD in a streaming way. These guidelines are encapsulated in a JSON-LD streaming document form, and a streaming RDF form. These forms, when they are detected, allow implementations to apply streaming optimizations.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.

This is an unofficial proposal.

This document was published by the JSON-LD Working Group as a First Public Working Group Note.

GitHub Issues are preferred for discussion of this specification. Alternatively, you can send comments to our mailing list. Please send them to public-json-ld-wg@w3.org (archives).

Please see the Working Group's implementation report.

Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the W3C Patent Policy. The group does not expect this document to become a W3C Recommendation.

This document is governed by the 1 March 2019 W3C Process Document.

1. Introduction

This document discusses the concerns on serializing and deserializing JSON-LD in a streaming manner. This document is primarily intended for the following audiences:

To understand the basics in this note you must first be familiar with JSON, which is detailed in [RFC8259]. You must also understand the JSON-LD syntax defined in JSON-LD 1.1 [JSON-LD11], which is the base syntax used for streaming processing. To understand how JSON-LD maps to RDF, it is helpful to be familiar with the basic RDF concepts [RDF11-CONCEPTS].

2. Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MAY, MUST, MUST NOT, SHOULD, and SHOULD NOT in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Streaming RDF Deserializers and Streaming RDF Serializers can claim conformance to this specification.

A conforming Streaming RDF Deserializer is a system that can deserialize JSON-LD to RDF for JSON-LD documents adhering to the Streaming Document Form as defined in this specification, and is a conforming RDF Deserializer according to the JSON-LD [JSON-LD11] specification minus the exceptions listed in this specification.

A conforming Streaming RDF Serializer is a system that can serialize RDF to JSON-LD for RDF datasets adhering to the streaming RDF dataset form as defined in this specification, and is a conforming RDF Serializer according to the JSON-LD [JSON-LD11] specification minus the exceptions listed in this specification.

The processing discussion in this specification merely contains implementation guidelines. Thus, Streaming RDF Deserializers and Streaming RDF Serializers may implement the algorithms given in this specification in any way desired, so long as the end result is indistinguishable from the result that would be obtained by the specification's algorithms.

Note

Implementers can partially check their level of conformance to this specification by successfully passing the test cases of the Streaming JSON-LD test suite. Note, however, that passing all the tests in the test suite does not imply complete conformance to this specification. It only implies that the implementation conforms to aspects tested by the test suite.

3. Streaming Document Form

There are multiple ways of describing data in JSON-LD, each having their own use cases. This section introduces a streaming JSON-LD document form, which enables JSON-LD documents to be deserialized in a streaming manner.

3.1 Importance of Key Ordering

The order in which key-value pairs occur in JSON-LD nodes conveys no meaning. For instance, the following two JSON-LD documents have the same meaning, even though they are syntactically different.

Example 1: Name, homepage and image come after @id
{
  "@context": "http://schema.org/",
  "@id": "https://www.rubensworks.net/#me",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg"
}
Example 2: Name, homepage and image come before @id
{
  "@context": "http://schema.org/",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg",
  "@id": "https://www.rubensworks.net/#me"
}

In a streaming JSON-LD document, the order of certain keys is important. This is because streaming JSON-LD processors may require the presence of some keys before others can be processed, and ordering keys in certain ways may lead to better processing performance.

In the two snippets before, the first example can be processed more efficiently by a streaming processor. Concretely, a streaming JSON-LD deserializer can emit an RDF triple each time a property ("name", "url", "image") has been read, because the "@id" has been defined before. This is because the "@id" defines the RDF subject, the property key defines the RDF predicate, and the property value defines RDF object. This ensures that all required information is needed for constructing and emitting an RDF triple each time a property is encountered.

For the second example, where the "@id" is only defined at the end, a streaming deserializer would have to buffer the properties until the "@id" key is encountered. Since the RDF subject of our triples is defined via "@id", the RDF triples can only be emitted after this last key has been read.

3.2 Required Key Ordering

In order for a JSON-LD document to be a in a streaming document form, the keys in each JSON node MUST be ordered according to the following order:

  1. @context
  2. @type
  3. Other properties

Each of these keys is optional, and may be omitted. Only those that are present must occur in the following order.

This order is important because @context can change the meaning of all following entries in the node and its children. Additionally, @type could indicate a type-scoped context, which may have the same implications of an @context. This means that these MUST always be processed before all other entries.

Note

Entries in nodes have a defined order when serialized as JSON. However, this this order is not always kept by JSON parsing libraries. This means that streaming processors MUST make use of JSON parsers that preserve this order to be effective.

Note

@type can be aliased to other keys, for which the order also applies.

3.4 Examples

This section is non-normative.

Hereafter, a couple of JSON-LD document examples are listed that either adhere, adhere with non-recommended order, or do not adhere to the streaming document form.

3.4.1 Valid Examples

Example 3: Valid streaming document with @context, @id, and other properties
{
  "@context": "http://schema.org/",
  "@id": "https://www.rubensworks.net/#me",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg"
}
Example 4: Valid streaming document with @context, blank @id, and other properties
{
  "@context": "http://schema.org/",
  "@id": "_:blank_node",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg"
}
Example 5: Valid streaming document with nested nodes
{
  "@context": "http://schema.org/",
  "@id": "https://www.rubensworks.net/#me",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg",
  "knows": {
    "@id": "https://greggkellogg.net/foaf#me",
    "name": "Gregg Kellogg"
  }
}
Example 6: Valid streaming document with nested nodes and embedded context
{
  "@context": "http://schema.org/",
  "@id": "https://www.rubensworks.net/#me",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg",
  "knows": {
    "@context": {
      "name": "http://xmlns.com/foaf/0.1/name"
    },
    "@id": "https://greggkellogg.net/foaf#me",
    "name": "Gregg Kellogg"
  }
}
Example 7: Valid streaming document with @context, @type-scoped context, @id, and other properties
{
  "@context": {
    "Person": {
      "@id": "http://schema.org/Person",
      "@context": "http://schema.org/"
    }
  },
  "@type": "Person",
  "@id": "https://www.rubensworks.net/#me",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg"
}

3.4.3 Invalid Examples

Example 10: Invalid streaming document where @context comes too late
{
  "@id": "https://www.rubensworks.net/#me",
  // @context must come before @id
  "@context": "http://schema.org/",
  "name": "Ruben Taelman",
  "url": "https://www.rubensworks.net/",
  "image": "https://www.rubensworks.net/img/ruben.jpg"
}
Example 11: Invalid streaming document where @type comes too late
{
  "http://schema.org/name": "Ruben Taelman",
  "@type": "http://schema.org/Person", // @type must come before properties
  "http://schema.org/url": {"@id": "https://www.rubensworks.net/"},
  "http://schema.org/image": {"@id": "https://www.rubensworks.net/img/ruben.jpg"}
}

3.5 Streaming Document Profile

JSON-LD documents can be signaled or requested in streaming document form. The profile URI identifying the streaming document form is http://www.w3.org/ns/json-ld#streaming.

The following example illustrates how this profile parameter can be used to request a streaming document over HTTP.

GET /ordinary-json-document.json HTTP/1.1
Host: example.com
Accept: application/ld+json;profile=http://www.w3.org/ns/json-ld#streaming

Requests the server to return the requested resource as JSON-LD in streaming document form.

4. Streaming RDF Form

This section introduces a streaming RDF dataset form, which enables RDF datasets to be processed in a streaming manner so that they can efficiently serialized into JSON-LD by a streaming JSON-LD processor.

4.1 Importance of Triple Ordering

The order in which RDF triples occur in an RDF dataset convey no meaning. For instance, the following two RDF datasets (serialized in RDF 1.1 Turtle [Turtle]) have the same meaning, even though they have a different order of triples.

Example 12: A first order of triples
@prefix schema: <http://schema.org/> .
<https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
<https://www.rubensworks.net/#me> schema:url <https://www.rubensworks.net/> .
<https://greggkellogg.net/foaf#me> schema:name "Gregg Kellogg" .
<https://greggkellogg.net/foaf#me> schema:url <https://greggkellogg.net/> .
Example 13: A second order of triples
@prefix schema: <http://schema.org/> .
<https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
<https://greggkellogg.net/foaf#me> schema:name "Gregg Kellogg" .
<https://www.rubensworks.net/#me> schema:url <https://www.rubensworks.net/> .
<https://greggkellogg.net/foaf#me> schema:url <https://greggkellogg.net/> .

For streaming JSON-LD processors, the order of RDF triples may be important. Processors that read triples one by one, and convert them to a JSON-LD document in a streaming manner, can benefit from having triples in a certain order.

For instance, the order from first snippet above can lead to more compact JSON-LD documents than the order from the second snippet when handled by a streaming JSON-LD processor. This is because the first order groups triples with the same subject, which can be exploited during streaming JSON-LD serialization by using the same "@id" key. The second order mixes subjects, which means that streaming JSON-LD serialization will have to assign separate "@id" keys for each triple, resulting in duplicate "@id" keys.

Streaming JSON-LD serializations of both examples can be seen below.

Example 14: More compact JSON-LD serialization of the first order
[
  {
    "@id": "https://www.rubensworks.net/#me",
    "http://schema.org/name": "Ruben Taelman",
    "http://schema.org/url": { "@id": "https://www.rubensworks.net/" }
  },
  {
    "@id": "https://greggkellogg.net/foaf#me",
    "http://schema.org/name": "Gregg Kellogg",
    "http://schema.org/url": { "@id": "https://greggkellogg.net/" }
  }
]
Example 15: Less compact JSON-LD serialization of the second order
[
  {
    "@id": "https://www.rubensworks.net/#me",
    "http://schema.org/name": "Ruben Taelman"
  },
  {
    "@id": "https://www.rubensworks.net/#me",
    "http://schema.org/name": "Gregg Kellogg"
  },
  {
    "@id": "https://greggkellogg.net/foaf#me",
    "http://schema.org/url": { "@id": "https://www.rubensworks.net/" }
  },
  {
    "@id": "https://greggkellogg.net/foaf#me",
    "http://schema.org/url": { "@id": "https://greggkellogg.net/" }
  }
]

4.3 Examples

This section is non-normative.

Hereafter, a couple of RDF datasets are listed, together with corresponding serialized JSON-LD in a streaming manner. Each example illustrates the importance of the recommended triple ordering within the streaming RDF dataset form.

The examples using named graphs are serialized in the RDF 1.1 TriG format [TriG].

4.3.1 @graph grouping

Example 16: Triples with the same named graph are grouped
@prefix schema: <http://schema.org/> .
<http://example.org/graph1> {
  <https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
}
<http://example.org/graph1> {
  <https://www.rubensworks.net/#me> schema:url <https://www.rubensworks.net/> .
}
<http://example.org/graph2> {
  <https://greggkellogg.net/foaf#me> schema:name "Gregg Kellogg" .
}
<http://example.org/graph2> {
  <https://greggkellogg.net/foaf#me> schema:url <https://greggkellogg.net/> .
}
Example 17: Triples with the same named graph can be grouped within the same @graph block
[
  {
    "@id": "http://example.org/graph1",
    "@graph": [
      {
        "@id": "https://www.rubensworks.net/#me",
        "http://schema.org/name": "Ruben Taelman",
        "http://schema.org/url": { "@id": "https://www.rubensworks.net/" }
      }
    ]
  },
  {
    "@id": "http://example.org/graph2",
    "@graph": [
      {
        "@id": "https://greggkellogg.net/foaf#me",
        "http://schema.org/name": "Gregg Kellogg",
        "http://schema.org/url": { "@id": "https://greggkellogg.net/" }
      }
    ]
  }
]

4.3.2 @id grouping

Example 18: Triples with the same subject are grouped
@prefix schema: <http://schema.org/> .
<https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
<https://www.rubensworks.net/#me> schema:url <https://www.rubensworks.net/> .
<https://greggkellogg.net/foaf#me> schema:name "Gregg Kellogg" .
<https://greggkellogg.net/foaf#me> schema:url <https://greggkellogg.net/> .
Example 19: Triples with the same subject can be grouped within the same @id block
[
  {
    "@id": "https://www.rubensworks.net/#me",
    "http://schema.org/name": "Ruben Taelman",
    "http://schema.org/url": { "@id": "https://www.rubensworks.net/" }
  },
  {
    "@id": "https://greggkellogg.net/foaf#me",
    "http://schema.org/name": "Gregg Kellogg",
    "http://schema.org/url": { "@id": "https://greggkellogg.net/" }
  }
]

4.3.3 Property grouping

Example 20: Triples with the same predicate are grouped
@prefix schema: <http://schema.org/> .
<https://www.rubensworks.net/#me> schema:name "Ruben" .
<https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
<https://www.rubensworks.net/#me> schema:url <https://www.rubensworks.net/> .
<https://www.rubensworks.net/#me> schema:url <https://github.com/rubensworks/> .
Example 21: Triples with the same predicate can be grouped within the same property array
[
  {
    "@id": "https://www.rubensworks.net/#me",
    "http://schema.org/name": [
      "Ruben",
      "Ruben Taelman"
    ],
    "http://schema.org/url": [
      { "@id": "https://www.rubensworks.net/" },
      { "@id": "https://github.com/rubensworks/" }
    ]
  }
]

4.3.4 @graph and @id grouping

Example 22: Statements about a named graph are group with triples within this named graph
@prefix schema: <http://schema.org/> .
<http://example.org/graph1> {
  <https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
}
<http://example.org/graph1> {
  <https://www.rubensworks.net/#me> schema:url <https://www.rubensworks.net/> .
}
<http://example.org/graph1> schema:name "Graph 1" .
<http://example.org/graph2> {
  <https://greggkellogg.net/foaf#me> schema:name "Gregg Kellogg" .
}
<http://example.org/graph2> {
  <https://greggkellogg.net/foaf#me> schema:url <https://greggkellogg.net/> .
}
<http://example.org/graph2> schema:name "Graph 2" .
Example 23: Statements about a named graph can be attached within the same @graph block
[
  {
    "@id": "http://example.org/graph1",
    "@graph": [
      {
        "@id": "https://www.rubensworks.net/#me",
        "http://schema.org/name": "Ruben Taelman",
        "http://schema.org/url": { "@id": "https://www.rubensworks.net/" }
      }
    ],
    "name": "Graph 1"
  },
  {
    "@id": "http://example.org/graph2",
    "@graph": [
      {
        "@id": "https://greggkellogg.net/foaf#me",
        "http://schema.org/name": "Gregg Kellogg",
        "http://schema.org/url": { "@id": "https://greggkellogg.net/" }
      }
    ],
    "name": "Graph 2"
  }
]

4.3.5 Subject and object grouping

Example 24: Triples about a subject come right after triples having this as object
@prefix schema: <http://schema.org/> .
<https://www.rubensworks.net/#me> schema:name "Ruben Taelman" .
<https://www.rubensworks.net/#me> schema:knows <https://greggkellogg.net/foaf#me> .
<https://greggkellogg.net/foaf#me> schema:knows "Gregg Kellogg" .
Example 25: Triples can be chained in nested nodes
[
  {
    "@id": "https://www.rubensworks.net/#me",
    "http://schema.org/name": "Ruben Taelman",
    "http://schema.org/knows": {
      "@id": "https://greggkellogg.net/foaf#me",
      "http://schema.org/name": "Gregg Kellogg"
    }
  }
]

5. Streaming Processing

Whenever a JSON-LD document is present in streaming document form, or if an RDF dataset is present in a streaming RDF dataset form, a processor MAY process these in a streaming manner.

This section describes high-level guidelines for processing JSON-LD in a streaming manner. Concretely, guidelines are given for deserializing JSON-LD to RDF, and serializing RDF to JSON-LD. Further details on processing can be found in JSON-LD 1.1 Processing Algorithms and API [JSON-LD11-API].

5.1 Deserialization

A streaming deserializer MAY be implemented by considering a JSON-LD document as a stream of incoming characters. By reading character-by-character, a deserializer can detect the contained JSON nodes and its key-value pairs.

A streaming deserializer MUST assume that the required key ordering of a streaming document is present. If a different order is detected, an error MUST be thrown with error code "invalid streaming key order".

The first expected entry in a node is @context. If such an entry is present, all following entries in this node can make use of it, possibly inheriting parts of the context from parent nodes. If such an entry is not present, only contexts from parent nodes are considered for this node.

If an @type entry (or any alias of @type) is detected, it is checked whether or not it defines a type-scoped context according to the current node's context. If this defines a type-scoped context, the context for the current node is overridden.
Additionally, the @type must emit rdf:type triples based on the current node's subject and values. This subject will possibly only be determined later on, which will require buffering of these incomplete triples.

Note

In case multiple type-scoped contexts apply, they must not be processed by order of appearance, but using the lexicographical order.

If an @id entry is detected, the RDF subject for the current node is defined for later usage. Any other entries that are detected before @id must be buffered until @id is found, or the node closes (which sets the subject to a fresh blank node).

For every other property, the default JSON-LD algorithms are followed based on the current node's subject.

As an example of a system architecture of a streaming JSON-LD deserializer can be found in this blog post.

5.2 Serialization

A streaming JSON-LD serializer reads triples one by one, and outputs a JSON-LD document character-by-character, which can be emitted in a streaming manner.
This MAY be a JSON-LD document in the streaming document form.

A streaming serializer can benefit from having triples ordered following a streaming RDF dataset form, but it SHOULD NOT assume that RDF datasets follow this form in full.

As a basis, a streaming serializer can produce an array of node objects or graph objects, each one representing a single RDF triple/quad.

On top of this base case, several optimizations can be applied to achieve a more compact representation in JSON-LD. These optimizations are dependent on the surrounding triples, which is determine by the overall triple order.

When a JSON-LD context is passed to a streaming serializer, compaction techniques MAY be applied. For instance, instead of writing properties as full IRIs, they can be compacted based on the presence of terms and prefixes in the context.

Due to the chained nature of RDF lists, serializing them to JSON-LD with the @list keyword in a streaming way may not always be possible, since you may not know beforehand if a triple is part of a valid RDF list. Optionally, a streaming RDF serializer MAY provide an alternative method to emit @list keywords.

Since streaming RDF processors process triples one by one, so that they don't need to keep all triples in memory, they loose the ability to deduplicate triples. As such, a streaming JSON-LD serializer MAY produce JSON-LD that contains duplicate triples.

A. References

A.1 Normative references

[JSON-LD11]
JSON-LD 1.1. Gregg Kellogg; Pierre-Antoine Champin; Dave Longley. W3C. 18 October 2019. W3C Working Draft. URL: https://www.w3.org/TR/json-ld11/
[JSON-LD11-API]
JSON-LD 1.1 Processing Algorithms and API. Gregg Kellogg; Dave Longley; Pierre-Antoine Champin. W3C. 17 April 2020. W3C Candidate Recommendation. URL: https://www.w3.org/TR/json-ld11-api/
[RDF11-CONCEPTS]
RDF 1.1 Concepts and Abstract Syntax. Richard Cyganiak; David Wood; Markus Lanthaler. W3C. 25 February 2014. W3C Recommendation. URL: https://www.w3.org/TR/rdf11-concepts/
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119
[RFC8174]
Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words. B. Leiba. IETF. May 2017. Best Current Practice. URL: https://tools.ietf.org/html/rfc8174
[RFC8259]
The JavaScript Object Notation (JSON) Data Interchange Format. T. Bray, Ed.. IETF. December 2017. Internet Standard. URL: https://tools.ietf.org/html/rfc8259

A.2 Informative references

[TriG]
RDF 1.1 TriG. Gavin Carothers; Andy Seaborne. W3C. 25 February 2014. W3C Recommendation. URL: https://www.w3.org/TR/trig/
[Turtle]
RDF 1.1 Turtle. Eric Prud'hommeaux; Gavin Carothers. W3C. 25 February 2014. W3C Recommendation. URL: https://www.w3.org/TR/turtle/