Abstract

This specification provides an API for representing binary and string data in web applications as a Stream object, as well as programmatically building and reading its contents. This includes:

This API is designed to be used in conjunction with other APIs and elements on the web platform, notably: File [FILE-API], XMLHttpRequest (e.g. with an overloaded send() method and response object for Stream objects) [XMLHTTPREQUEST2], postMessage, and Web Workers [WEBWORKERS].

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is not complete. It is subject to major changes and, while early experimentations are encouraged, it is therefore not intended for implementation.

This document was published by the W3C Web Applications (WebApps) as a Working Draft. This document is intended to become a W3C Recommendation. If you wish to make comments regarding this document, please send them to public-webapps@w3.org (subscribe, archive) with a Subject prefix of [streams-api]. If you wish to submit a bug, please use Bugzilla. All comments and bug reports are welcome.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

1. Introduction

This section is non-normative.

Web applications should have the ability to acquire and manipulate data in a wide variety of forms, including as a sequence of data made available over time. This specification defines the basic representation for Streams, errors raised by Streams, and programmatic ways to create, read, and write to Streams.

The Stream interface represents binary data which can be obtained over time and read once. A Stream can come from API producers such as XMLHttpRequest, or can be built using the Stream constructor.

The Stream interface provides a read method for reading the data from a Stream as a StreamConsumeResult, which provides the data as a Blob, ArrayBuffer, or as DOMString, and should happen asynchronously on the user agent’s main thread. Additionally, the stream can also be used in API consumers such as a media element.

The Stream interface also provides a write method for writing data to a Stream as a Blob, ArrayBuffer, or as DOMString, and should happen asynchronously on the user agent’s main thread.

An asynchronous API for reading Streams prevents blocking and UI “freezing” on a user agent’s main thread. This specification defines an asynchronous API to access a Stream. Error conditions that may arise during reading of a Stream will be handled by a reject callback set to the promise returned by the read() method. An example will be illustrative.

In the example below, different code blocks handle progress, error, and success conditions. The example demonstrates how to read a chunk of data from a Stream using read. The Stream may of come from a producer such as XMLHttpRequest. Additionally, it demonstrates how to read a stream until an EOF is encountered.

Example 1
// Read the first 1024 bytes of the stream as UTF-8
stream.readEncoding = "UTF-8";
stream.readType = "arraybuffer";
Promise readPromise = stream.read(1024);
readPromise.then(
  function(result) {
    console.log("Loaded" + result.size + " bytes");
    // Handle result.data
  },
  function(error) {
    // Handle error
  }
);

// Read data from the stream repeatedly
function readUntilEof() {
  stream.read(1024).then(
    function(result) {
      // Handle read data
      someProcessFunction(result.data);

      // Print progress
      someReportFunction(result.size);

      if (!result.eof) {
        readUntilEof();
      }
    },
    function(error) {
      // Handle error
    }
  );
}

In the example below, different code blocks handle progress, error, and success conditions. The example below demonstrates how to obtain a Stream from XMLHttpRequest to begin playing a large video in readystate 3. The example takes the Stream from a producer, XMLHttpRequest, and gives to a consumer, the video tag.

Example 2
function handler() {
  if(this.readyState == this.LOADING) {
    var theStream = this.response;
    var streamURL = URL.createObjectURL(theStream);
    document.getElementById("myVideoTag").src = streamURL;
  }
}

var client = new XMLHttpRequest();
client.onreadystatechange = handler;
client.setRequestHeader('customHeader', 'value');
client.setRequestHeader('customHeader2', 'value2');
client.open("GET", "myvideo.h264");
client.responseType = "stream";
client.send();

In addition to reading a Stream, this specification introduces a programatic way to write data to a Stream. The Stream interface provides a write() method which allows applications to build the data to be read by a Stream consumer by appending the data to an internal buffer. write() supports appending data as a Blob, ArrayBuffer, or DOMString to the buffer.

The example below demonstrates how to use write() to load a Stream into the audio tag, whose data could be processed and built dynamically at read time.

Example 3
var theStream = new Stream("audio/mp3");

function writeData(){
  var moreData;
  // Do work to create more data to place into the stream

  // If we have no more data to process and place in the stream, we close
  if (moreData == null){
    theStream.close();
  } else{
    theStream.write(moreData).then(
      function () {
        writeData();
      },
      function (error) {
        // Handle error
      }
    );
  }
}

var streamURL = URL.createObjectURL(theStream);
document.getElementById('audioTag').src = streamURL;

writeData();

2. Streams

This section introduces the Stream interface, as well as accompanying interfaces required as part of the Stream implementation. This includes a constructor to build a Stream, as well as methods to read, write, skip, pipe, and close streams.

2.1 The Stream Interface

This interface represents a raw sequence of linear data which can be read only once over time. It provides a type attribute which represents the type of data in the Stream. It also provides the ability to read and write the contents of the Stream.

A Stream is an object that:

A Stream object has an associated write closed flag. It is set when the close() method is called. This flag is internal, so scripts cannot access it directly.

A Stream object has an associated write pending flag. It is set when the write() method is called and unset when it's completed. This flag is internal, so scripts cannot access it directly.

A Stream object has an associated read pending flag. It is set when read(), skip() or pipe() method is called and unset when it's completed. This flag is internal, so scripts cannot access it directly.

A Stream holds a sequence of bytes possibly terminated by a terminator. Writing bytes to a Stream means appending the bytes to the sequence. Terminating a Stream means appending a terminator to the sequence. Reading bytes from a Stream pops bytes from the head of the sequence. If a terminator is encountered while reading bytes from a Stream, it is said the EOF is reached. This sequence is internal, so scripts cannot access it directly. 

[ Constructor (optional DOMString type)]
interface Stream {
    readonly    attribute DOMString      type;
                attribute StreamReadType readType;
                attribute DOMString      readEncoding;
    Promise write ((DOMString or ArrayBufferView or Blob) data);
    void    close ();
    Promise read (optional [Clamp] unsigned long long size);
    Promise skip (![Clamp] unsigned long long size);
    Promise pipe (!(Stream or Stream[]) destination, optional [Clamp] unsigned long long size);
};

2.1.1 Constructors

Stream
Constructs a Stream and sets the type to the specified value.
ParameterTypeNullableOptionalDescription
type✘✔ Specifies the MIME type [RFC2046] of the Stream.

2.1.2 Attributes

readEncoding of type DOMString,

A DOMString that represents the label of an encoding [EncodingDetermination]. If set, it will be used as part of the encoding determination used when processing a read call. If not set, it will return the empty string.

readType of type StreamReadType,

Returns the type of the last read operation taken on the Stream. On getting, conforming user agents must return the type of the last read operation. If no read operation has taken place and the readType was not set, then return the empty string. This can be set to the empty string (default), "arraybuffer", "blob" and "text" to change the type of the read operation.

type of type DOMString, readonly
Returns the ASCII-encoded string in lower case representing the media type of the Stream, expressed as an RFC2046 MIME type [RFC2046]. Conforming user agents SHOULD return the MIME type of the Stream, if it is known. If conforming user agents cannot determine the media type of the Stream, they MUST return the empty string. A string is a valid MIME type if it matches the media-type token defined in section 3.7 "Media Types" of RFC 2616 [HTTP11].

2.1.3 Methods

close

This method closes the Stream and does not allow any future writes. This is an irreversible operation; once a Stream has been closed, it cannot be written to again. When all data has been read from the Stream on which close() has been called, i.e. EOF is reached, it resolves the Promise returned by read() with a StreamConsumeResult with the eof attribute set to true. The user agent must run the steps below:

  1. If the Stream has been neutered, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  2. If the write closed flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  3. If the write pending flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  4. Set the write closed flag.
  5. Terminate the Stream.
  6. If an error has occurred during writing a stream termination, neuter the Stream and terminate these steps.
No parameters.
Return type:
pipe

This method transfers data from the Stream to another Stream. This method takes a destinations and optionally a size. Another read(), skip() or pipe() call must not be made until the returned Promise is resolved or rejected. Resolution of the returned Promise doesn't necessarily mean that the data transferred to the destination Stream has been successfully read from the Stream. The user agent must run the steps below:

  1. If the Stream has been neutered, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  2. If the read pending flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  3. If size is specified but is 0, throw a "SyntaxError" [DOM4] exception and terminate these steps.
  4. If destinations is a Stream, let destinations instead be an array consisting of just that Stream.
  5. Set the read pending flag.
  6. Let readPromise be a new promise.
  7. Return the pipe() method with readPromise, but continue to process the steps in this algorithm.
  8. If size is specified, read data from the stream until size bytes are read.
  9. Otherwise, read data from the Stream until EOF is reached.
  10. As read data becomes available, write newly read data to destinations.
  11. If any error has occurred during reading or writing to destinations, neuter the Stream, let exception be an "InvalidStateError" exception and run Reject(readPromise, exception) as specified in the promises spec and terminate these steps.
  12. Once read and write are both completed for all destination streams, run the following algorithm:
    1. Let result be a newly created StreamConsumeResult object.
    2. If EOF is reached, set eof attribute of result to true.
    3. Otherwise, set eof attribute of result set to false.
    4. Set size attribute of result to the total size of the read data.
    5. Unset the read pending flag and run Resolve(readPromise, result) as specified in the promises spec.
ParameterTypeNullableOptionalDescription
destination✘✘Destination Stream.
size✘✔Number of bytes to transfer.
Return type:
read

This method reads data from the Stream. This method takes an optional size which represents the number of bytes to be read. Another read(), skip() or pipe() call must not be made until the returned Promise is resolved or rejected. The user agent must run the steps below (unless otherwise indicated):

  1. If the Stream has been neutered, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  2. If the read pending flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  3. If size is specified but is 0, throw a "SyntaxError" [DOM4] exception and terminate these steps.
  4. Set the read pending flag.
  5. Let readPromise be a new promise.
  6. Return readPromise, but continue to process the steps in this algorithm.
  7. If size is specified, read data from the Stream until size bytes are read.
  8. Otherwise, read data from the Stream until any non-zero bytes are read.
  9. If an error has occurred during reading, neuter the Stream, let exception be an "InvalidStateError" [DOM4] exception and run Reject(readPromise, exception) as specified in the promises spec and terminate these steps.
  10. Let result be a newly created StreamConsumeResult.
  11. If EOF is reached, set the eof attribute of result to true.
  12. Otherwise, set the eof attribute of result to false.
  13. Set the data attribute of result to the result of executing the steps below.
    If readType is the empty string or "text"
    1. If readEncoding is not null, let charset be readEncoding.
    2. Otherwise, let charset be utf-8.
    3. Let result be result of decoding the data read using fallback encoding charset.
    If readType is the empty string or "blob"
    Let result be a blob created from the read data
    If readType is the empty string or "arraybuffer"
    Let result be an array buffer created from the read data
  14. Unset the read pending flag and run Resolve(readPromise, result) as specified in the promises spec.
ParameterTypeNullableOptionalDescription
size✘✔Number of bytes to read.
Return type:
skip

This method reads data from the Stream and ignore them. This method takes optionally a size which represents the number of bytes to be read and ignored. Another read(), skip() or pipe() call must not be made until the returned Promise is resolved or rejected. The user agent must run the steps below:

  1. If the Stream has been neutered, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  2. If the read pending flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  3. If size is specified but is 0, throw a "SyntaxError" [DOM4] exception and terminate these steps.
  4. Set read pending flag.
  5. Let readPromise be a new promise.
  6. Return the skip() method with readPromise, but continue to process the steps in this algorithm.
  7. If size is specified, read data from the Stream until size bytes are read.
  8. Otherwise, read data from the Stream until any non-zero bytes are read.
  9. If any error has occurred during reading, neuter the Stream, let exception be an "InvalidStateError" [DOM4] exception and run Reject(readPromise, exception) as specified in the promises spec and terminate these steps.
  10. Let result be a newly created StreamConsumeResult object.
  11. If EOF is reached, set the eof attribute of result to true.
  12. Otherwise, set the eof attribute of result set to false.
  13. Set the size attribute of result to the size of the read data in bytes.
  14. Unset the read pending flag and run Resolve(readPromise, result) as specified in the promises spec.
ParameterTypeNullableOptionalDescription
size✘✘Number of bytes to read and ignore.
Return type:
write

This method writes data to the Stream. Another write() or close() call must not be made until the returned Promise is resolved or rejected. Resolution of the returned Promise doesn't necessarily mean that the data written has been successfully read. The user agent must run the steps below (unless otherwise indicated):

  1. If the Stream has been neutered, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  2. If the write closed flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  3. If the write pending flag is set, throw an "InvalidStateError" [DOM4] exception and terminate these steps.
  4. Set the write pending flag.
  5. Let writePromise be a new promise.
  6. Return writePromise, but continue to process the steps in this algorithm.
  7. Execute the rules below, depending on the type of data:
    ArrayBufferView
    Let rawData be the raw data represented by the Blob object.
    Blob
    Let rawData be the data stored in the section of the buffer described by the ArrayBuffer object that the ArrayBufferView object references.
    DOMString
    Let rawData be the result of encoding data to binary data using the encoding determined by the [EncodingDetermination]
  8. Write rawData to the Stream.
  9. If an error has occurred during the write, neuter the Stream, let exception be an "InvalidStateError" [DOM4] exception and run Reject(writePromise, exception) as specified in the promises spec and terminate this algorithm.
  10. Unset the write pending flag and run Resolve(writePromise, undefined) as specified in the promises spec. Implementations may delay this step if appropriate.
ParameterTypeNullableOptionalDescription
data✘✘Data to write.
Return type:

2.2 StreamConsumeResult Interface

interface StreamConsumeResult {
    readonly    attribute boolean            eof;
    readonly    attribute any                data;
    readonly    attribute unsigned long long size;
};

2.2.1 Attributes

data of type any, readonly
The contents of the read request
eof of type boolean, readonly
specifies if the given read resulted in an EOF for the Stream
size of type unsigned long long, readonly
The size, in bytes, of the data read

2.3 StreamReadType Interface

enum StreamReadType {
    "blob",
    "arraybuffer",
    "text"
};
Enumeration description
blobRead operations should return data as a Blob
arraybufferRead operations should return data as an ArrayBuffer
textRead operations should return data as a DOMString

2.4 URIs for Stream

To reference a Stream, the same URI used for Blobs and Files in 6.7. A URI for Blob and File reference of the File API specification should be used. [FILE-API] The definitions of Origin, Lifetime, Referencing, and Dereferencing of a Blob should be applied to a Stream.

2.4.1 Creating and Revoking a Stream URI

A Stream URI is a Blob URI that is referencing a Stream. These URIs are created and revoked using methods exposed on the URL object, as defined in 6.7.5. Creating and Revoking a Blob URI of the File API specification. [FILE-API]

URL.createObjectURL and URL.revokeObjectURL should both be extended as follows: 

interface URL {
    static DOMString? createObjectURL ((Blob or Stream) object);
    static DOMString? createFor ((Blob or Stream) object);
    static void       revokeObjectURL (DOMString url);
};
2.4.1.1 Methods
createFor, static

The extension onto createFor should have the following steps added.

Returns a unique Blob URL each time it is called on a valid object argument, which is a non-null Stream in scope of the global object's URL property from which this static method is called. Blob URLs created with this method are said to be auto-revoking since user-agents are responsible for the revocation of Blob URLs created with this method, subject to the lifetime stipulation for Blob URLs. This method must act as follows:

  1. If called with a Stream argument that is NOT valid, then user agents must return null.
  2. If called with a valid Stream argument, user agents must run the following sub-steps:
    1. If the read pending flag of the Stream is set, return null.
    2. Set the read pending flag of the Stream.
    3. Return a unique Blob URI that can be used to dereference the stream argument.
    4. Add an entry to the Blob URL Store for this Blob URL.
    5. Add an entry in the Revocation List for this Blob URL.
ParameterTypeNullableOptionalDescription
object✘✘
Return type: , nullable
createObjectURL, static

The extension onto createObjectURL should have the following steps added.

Returns a unique Blob URL each time it is called on a valid object argument, which is a non-null Stream in scope of the global object's URL property from which this static method is called. This method must act as follows:

  1. If called with a Stream argument that is NOT valid, then user agents must return null.
  2. If called with a valid Stream argument, user agents must run the following sub-steps:
    1. If the read pending flag of the Stream is set, return null.
    2. Set the read pending flag of the Stream.
    3. Return a unique Blob URI that can be used to dereference the stream argument.
    4. Add an entry to the Blob URL Store for this Blob URL.
ParameterTypeNullableOptionalDescription
object✘✘
Return type: , nullable
revokeObjectURL, static

The extension onto revokeObjectURL should have the following steps added.

  1. If the URL refers to a Blob or Stream that is both valid and in the same origin of the global object’s URL property on which this static method was called, user agents MUST return a 404 response code when the URL is dereferenced.
  2. If the URL refers to a Blob or Stream that is not valid or if the value provided for the URL argument is not a Blob URI or if the URL argument refers to a Blob or Stream that is not in the same origin as the global object’s URL property, this method call does nothing. User agents MAY display a message on their error console.
ParameterTypeNullableOptionalDescription
url✘✘
Return type:

3. Stream Consumers and Producers

Streams can be both produced and consumed by various APIs. APIs which create streams are identified as producers, and ones which read and act on a stream are known as consumers. This section identifies some of the APIs where Streams may be produced and consumed.

Note
The list of producers and consumers below is not an exhaustive list. It is placed here as informative for the time being.

3.1 Stream Consumers

This section outlines APIs which can consume a Stream object

3.2 Stream Producers

This section outlines APIs which can produce a Stream object

4. Security Considerations

A Stream should have the same security considerations as a Blob. This is outlined in 6.8. Security Considerations of the File API specification. [FILE-API] Because a Stream uses a Blob URI, cross origin requests on a Stream will not be supported.

5. Extension of XMLHttpRequest

This specification proposes an extension to XMLHttpRequest [XMLHTTPREQUEST2] to add support for Stream. This section is temporary and is meant to provide a recommendation for how Stream should be incorporated into XMLHttpRequest. This will extend XMLHttpRequest to allow for receiving and uploading of a Stream. One such scenario is providing access to data during readyState 3 (LOADING). The sections below document in detail what extensions must be done to XMLHttpRequest to support Stream.

5.1 Addition of stream response entity body

The section named "Response Entity Body" in XMLHttpRequest specification [XMLHTTPREQUEST2] should have the following additions:

The stream response entity body is either a Stream representing the response entity body or null. If the stream response entity body is null, let it be the return value of the following algorithm:

  1. If the response entity body is null, return an empty Stream object.
  2. Return a Stream object representing the response entity body.

5.2 Addition of "stream" responseType

A new value for the responseType attribute "stream" should be introduced.

In the IDL list in the section named "Interface XMLHttpRequest" in XMLHttpRequest specification [XMLHTTPREQUEST2], the definition of XMLHttpRequestResponseType enum should now read: 

enum XMLHttpRequestResponseType {
  "",
  "arraybuffer",
  "blob",
  "stream",
  "document",
  "json",
  "text"
}

5.3 Modification on the response attribute

The algorithm of the response attribute should be modified to handle the new responseType value "stream". A Stream is binary data obtained sequentially over time. Given this, a Stream should be accessible in readyState 3 (LOADING).

The section named "The response attribute" in XMLHttpRequest specification [XMLHTTPREQUEST2] should now read:

The response attribute must return the result of running these steps:

If responseType is the empty string or "text"
The same as the original XMLHttpRequest specification.
If responseType is "stream"
  1. If the state is not LOADING or DONE, return null.
  2. If the error flag is set, return null.
  3. Return the stream response entity body.
Otherwise
The same as the original XMLHttpRequest specification.

5.4 send()

The switch in otherwise case of step 4 of The section named "The send() method" in XMLHttpRequest specification [XMLHTTPREQUEST2] should have the following additions:

Stream

If the object's type attribute is not the empty string let mime type be its value.

Set the read pending flag for the stream.

Let the request entity body be the raw data represented by data.

Once the read is completed for the request, call close() on the stream

6. Requirements and Use Cases

The Stream type allows for completion of several end-to-end experiences. This section covers what the requirements are for this API, and illustrates some use cases.

A. Acknowledgements

Thanks to Eliot Graff for editorial assistance. Special thanks to the W3C. The editor would like to thank Anne van Kesteren, Austin William Wright, Aymeric Vitte, Domenic Denicola, Isaac Schlueter, Jonas Sicking, Kenneth Russell, Yusuke Suzuki, Adrian Bateman for their contributions to this specification.

B. References

B.1 Normative references

[DOM4]
Anne van Kesteren; Aryeh Gregor; Lachlan Hunt; Ms2ger. DOM4. 6 December 2012. W3C Working Draft. URL: http://www.w3.org/TR/dom/
[EncodingDetermination]
Anne van Kesteren; Joshua Bell. Encoding. URL: http://encoding.spec.whatwg.org/
[FILE-API]
Arun Ranganathan; Jonas Sicking. File API. 12 September 2013. W3C Last Call Working Draft. URL: http://www.w3.org/TR/FileAPI/
[HTTP11]
R. Fielding et al. Hypertext Transfer Protocol - HTTP/1.1. June 1999. RFC. URL: http://www.ietf.org/rfc/rfc2616.txt
[RFC2046]
N. Freed; N. Borenstein. Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types (RFC 2046). November 1996. RFC. URL: http://www.ietf.org/rfc/rfc2046.txt
[WEBWORKERS]
Ian Hickson. Web Workers. 1 May 2012. W3C Candidate Recommendation. URL: http://www.w3.org/TR/workers/
[XMLHTTPREQUEST2]
Anne van Kesteren. XMLHttpRequest Level 2. 16 August 2011. W3C Working Draft. URL: http://www.w3.org/TR/XMLHttpRequest2/