This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 28496 - Allow Blob constructor to take ownership of ArrayBuffer(View) / invoke DetachArrayBuffer
Summary: Allow Blob constructor to take ownership of ArrayBuffer(View) / invoke Detach...
Status: NEW
Alias: None
Product: WebAppsWG
Classification: Unclassified
Component: File API (show other bugs)
Version: unspecified
Hardware: PC Linux
: P2 normal
Target Milestone: ---
Assignee: Arun
QA Contact: public-webapps-bugzilla
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-15 22:43 UTC by Andrew Sutherland
Modified: 2015-10-19 16:37 UTC (History)
4 users (show)

See Also:


Attachments

Description Andrew Sutherland 2015-04-15 22:43:52 UTC
Motivation: For the Firefox OS email app when we download attachments we end up with a lot of TypedArray instaces piling up that exist only to be wrapped into a Blob and then forgotten.  Lacking any way to neuter ArrayBuffers directly, it would be nice if the Blob could takeownership of them as we pass them in.

Arguably it could be considered a common data-flow to create some data with a TypedArray and then stuff it in a Blob to store it somewhere

Risks:
- I guess the 2-step process of "new Blob(takingOwnership).close()" could end up as a disturbingly hacky general purpose means of disposing of the contents of TypedArrays.

Mitigation strategies:
- In our specific case we are using the proprietary-mozTCPSocket that just throws typed arrays at us, but I understand the Streams API provides a mechanism to allow reads into existing buffers.  APIs that allow use of existing buffers won't suffer from the "oh no, so many ArrayBuffers!" problem.
- GC will catch the stuff eventually.  Just crank up the GC.
Comment 1 Domenic Denicola 2015-04-15 23:04:45 UTC
Out of curiosity why do you want the data to be in blob form if you already have it in typed array form?
Comment 2 Andrew Sutherland 2015-04-15 23:28:02 UTC
(In reply to Domenic Denicola from comment #1)
> Out of curiosity why do you want the data to be in blob form if you already
> have it in typed array form?

For storage and resource consumption purposes:
- Firefox OS's (nonstandard) API to write to the sdcard, DeviceStorage, takes a Blob (https://developer.mozilla.org/en-US/docs/Web/API/DeviceStorage/addNamed).  - Gecko's IndexedDB implementation currently stores Blobs on the filesystem but TypedArrays directly inside the SQLite database.  (Noting that this will change so that when the structure clone payload is large, it will be stored on the filesystem too: https://bugzil.la/964561)
- The attachments are effectively opaque blobs of data to the email app anyways, if we've stored the attachment in IndexedDB, when we load the record for a mail body, it's a lot nicer to have a Blob handle than the entire contents of the TypedArray loaded directly into memory.  (Obviously one could use a separate key/value for that, but a handle is still more useful because...)
- For display purposes, etc., Blobs/Files are sent to other apps (using non-standard https://developer.mozilla.org/en-US/docs/Web/API/Web_Activities).
Comment 3 Domenic Denicola 2015-04-16 00:26:15 UTC
Hmm, unfortunately it sounds like the use cases here only arise as a result of nonstandard APIs or implementation choices. Not exactly a clear-cut argument for adding a new platform feature...
Comment 4 Andrew Sutherland 2015-04-16 00:38:40 UTC
I might have erred too much on the side of describing our specific use-case, so I'll step back from that, but it's worth noting that these are issues facing all JS code trying to do app-y things, even if different nonstandard APIs are used.  For example, Google Chrome's 2nd nonstandard TCP API likewise provides an ArrayBuffer with no possibility of using readInfo semantics: https://developer.chrome.com/apps/sockets_tcp#event-onReceive

I think the argument would be analogous to that for the addition of close() to Blob in the first place.  Jonas Sicking made the argument in bug 16952 comment 2:

> Blobs represent a potentially large amount of data, which in best case is on
> disk, worst case in memory, in both cases expensive on mobile.
> 
> It's generally a bad idea to rely on GC for managing expensive resources. (I
> believe that is a quote from Darin Fisher, but don't quote me on that :) ).

Analogously, if you create a memory-backed Blob and you can get rid of the Blob, why can't you get rid of its constituents, or ideally (as requested here), transfer ownership to the Blob in the first case.

ni'ing Jonas for his thoughts here.
Comment 5 Andrew Sutherland 2015-04-16 00:41:06 UTC
(In reply to Andrew Sutherland from comment #4)
> with no possibility of using readInfo semantics

s/readInfo/readInto/, as in use an existing ArrayBuffer to hold the received TCP payload.
Comment 6 Jonas Sicking (Not reading bugmail) 2015-06-05 22:31:41 UTC
I don't see anything FirefoxOS specific here.

If you've done some in-memory processing on a big chunk of data, you'll likely have that data in the form of an ArrayBuffer.

If you then want to write that data to IndexedDB you probably want to write it in the form of a Blob. This is especially true for IndexedDB implementations which optimize storage of large Blobs in IndexedDB (hopefully all of them).

The only way to do that is by doing |new Blob([arraybuffer])|. However that operation requires that the memory buffer is copied. This is both bad for performance and doubles the amount of memory used.

Technically the UA could implement some form of copy-on-write scheme for the arraybuffer data. However I think that is very hard to do without sacrificing performance of all arraybuffer operations, which is unlikely something we want to do. This is why transferring of arraybuffers were invented in the first place.


Adding a Blob constructor which allows declaring which ArrayBuffers should be transferred into the Blob seems like a simple addition to the platform.
Comment 7 Arun 2015-06-07 07:08:16 UTC
(In reply to Jonas Sicking from comment #6)
> I don't see anything FirefoxOS specific here.
> 
> If you've done some in-memory processing on a big chunk of data, you'll
> likely have that data in the form of an ArrayBuffer.
> 
> If you then want to write that data to IndexedDB you probably want to write
> it in the form of a Blob. This is especially true for IndexedDB
> implementations which optimize storage of large Blobs in IndexedDB
> (hopefully all of them).
> 
> The only way to do that is by doing |new Blob([arraybuffer])|. However that
> operation requires that the memory buffer is copied. This is both bad for
> performance and doubles the amount of memory used.
> 
> Technically the UA could implement some form of copy-on-write scheme for the
> arraybuffer data. However I think that is very hard to do without
> sacrificing performance of all arraybuffer operations, which is unlikely
> something we want to do. This is why transferring of arraybuffers were
> invented in the first place.
> 
> 
> Adding a Blob constructor which allows declaring which ArrayBuffers should
> be transferred into the Blob seems like a simple addition to the platform.


I understand why we should do this, and why, despite the fact that a copy-on-write scheme would keep the constructor as is, it is probably untenable.

I suppose we could have an ArrayBuffers-To-Be-Transferred type of dictionary argument. But instead of putting it in the constructor, might we not have something like a transferrableList() method (name TBD) that transfers ownership (and perhaps appends)?

https://w3c.github.io/FileAPI/#dfn-Blob
Comment 8 Jonas Sicking (Not reading bugmail) 2015-06-07 17:32:51 UTC
I think we definitely want to keep Blob's as immutable readonly objects. I don't think we should allow appending to an existing Blob.

So I think *some* form of constructor syntax is needed. But I don't have any concrete proposals.
Comment 9 Anne 2015-06-09 05:24:31 UTC
Presumably new Blob() takes a dictionary somewhere. We should just copy the transferList thingie from postMessage() (but then as dictionary member rather than new argument).
Comment 10 Jonas Sicking (Not reading bugmail) 2015-06-15 19:37:46 UTC
Yeah, something like:

new Blob([myArrayBuffer], { transfer: [myArrayBuffer] });
and
new File([otherBuffer], "myFile", { transfer: [otherBuffer] });

would work quite well I think.
Comment 11 Arun 2015-10-19 16:37:33 UTC
This is now https://github.com/w3c/FileAPI/issues/23 .