Our use case for this with Peergos is downloading or streaming large encrypted files which are reassembled and decrypted in the browser.

For an example see the following. This opens a 120MB version of Sintel and streams it to a video element. In Chrome this is done 5MiB at a time so it can start much faster than in Firefox where it has to fallback to loading the entire video into RAM and decrypting (the keys are in the url hash fragment) before it can start playing:

https://demo.peergos.net/#pQd8rmrEhBN1XBodCnuhegoEumwDbiFEjkYongNGkhRSAk1anF74Yv4t/pQd8rmrEhBN1PqDwhTEcvK5ezKzQ1UWFPmehpE146BxZpC6vHdrAnmZv/CX9xfVdKF7tPW2LRjz3RvoePmwAsNVsTAkAftgrbqKhW/5Pf7SvvCW9dLjFaQB5GXRHuQ5HZU6cjpkVU7vxzzLtgmPt6ULV9?streaming=true&open=true

This means that currently in Firefox we are limited to loading files that will fit into ram, which means we have to recommend Chrome. We would love to be able to recommend Firefox given our privacy focus.

(In reply to ianopolous from comment #2)

Our use case for this with Peergos is downloading or streaming large encrypted files which are reassembled and decrypted in the browser.

For an example see the following. This opens a 120MB version of Sintel and streams it to a video element. In Chrome this is done 5MiB at a time so it can start much faster than in Firefox where it has to fallback to loading the entire video into RAM and decrypting (the keys are in the url hash fragment) before it can start playing:

https://demo.peergos.net/#pQd8rmrEhBN1XBodCnuhegoEumwDbiFEjkYongNGkhRSAk1anF74Yv4t/pQd8rmrEhBN1PqDwhTEcvK5ezKzQ1UWFPmehpE146BxZpC6vHdrAnmZv/CX9xfVdKF7tPW2LRjz3RvoePmwAsNVsTAkAftgrbqKhW/5Pf7SvvCW9dLjFaQB5GXRHuQ5HZU6cjpkVU7vxzzLtgmPt6ULV9?streaming=true&open=true

This means that currently in Firefox we are limited to loading files that will fit into ram, which means we have to recommend Chrome. We would love to be able to recommend Firefox given our privacy focus.

I'm curious on why you would need Writeable Stream to do this.

There are lots of Web sites having that exact requirements, none of them using writeable streams.

(In reply to Jean-Yves Avenard [:jya] from comment #3)

(In reply to ianopolous from comment #2)

Our use case for this with Peergos is downloading or streaming large encrypted files which are reassembled and decrypted in the browser.

For an example see the following. This opens a 120MB version of Sintel and streams it to a video element. In Chrome this is done 5MiB at a time so it can start much faster than in Firefox where it has to fallback to loading the entire video into RAM and decrypting (the keys are in the url hash fragment) before it can start playing:

https://demo.peergos.net/#pQd8rmrEhBN1XBodCnuhegoEumwDbiFEjkYongNGkhRSAk1anF74Yv4t/pQd8rmrEhBN1PqDwhTEcvK5ezKzQ1UWFPmehpE146BxZpC6vHdrAnmZv/CX9xfVdKF7tPW2LRjz3RvoePmwAsNVsTAkAftgrbqKhW/5Pf7SvvCW9dLjFaQB5GXRHuQ5HZU6cjpkVU7vxzzLtgmPt6ULV9?streaming=true&open=true

This means that currently in Firefox we are limited to loading files that will fit into ram, which means we have to recommend Chrome. We would love to be able to recommend Firefox given our privacy focus.

I'm curious on why you would need Writeable Stream to do this.

There are lots of Web sites having that exact requirements, none of them using writeable streams.

Can you name one? What technique do they use? The closest I'm aware of is Mega, and they have the exact same problem. In Chrome they solve it with a Chrome specific file system api, and in Firefox they resort to loading it into RAM, as I've suggested, and downloading from a blob.

The general problem is for large files that are generated in the browser to be streamed directly to a host file (or or video tag). We generate them by reassembling fragments and decrypting in the browser with keys that are not exposed to the server.

P.S. N.B. the link I gave longer works - we are in middle of changing some things. I'll post another link when we're done.

As promised, here is an updated link to streaming Sintel using writable streams if they are present. This can start within a few seconds in Chrome, but in Firefox has to wait until the entire 120mb video is downloaded and decrypted (and it may crash if it doesn't fit in ram):

https://demo.peergos.net/#6MDZhRRPT4ugkJuUfcPPhf1US9u7FvRALmj42mJ6e3yDibnLtqfhchE6Frm6Lf/6MDZhRRPT4ugkJuUfcZdxu6JLKyrLBE36Kasxb4jix7An4dbeiekpDF6h2fDBM/G5KGmeQzGbaA16NGXRR6MtzkTw54YABd8wxaYxZKecth/5Pf7SvriCWAvPWM2ubDhCXbFasZFmPTZEyjHLKJKa26tYxCxN1R?open=true

Can an update be provided on this ticket?
I can see that writeable streams has been in nightly for a number of releases. The release build has functionality behind javascript.options.writable_streams, which when enabled works for our purposes.
Is it expected to be enabled by default soon?
In Firefox 74 I have noticed that the statement:
'serviceWorker' in navigator.
Is now also returning false in the release build; nightly returns true.

I don't plan to enable this in nightly (and then forward into releases) until bug 1502355 is fixed. From what people have been telling us, generally writable streams only is not what most people want, they also want piping at the same time.

Moreover, the number of web-platform-tests for writable streams is not particularly comprehensive enough, for me to really want to trust shipping what we've implemented. I'd prefer us to have the additional pounding that piping will afford us, before I flip this on.

(In reply to Jeff Walden [:Waldo] from comment #8)

I don't plan to enable this in nightly (and then forward into releases) until bug 1502355 is fixed. From what people have been telling us, generally writable streams only is not what most people want, they also want piping at the same time.

Moreover, the number of web-platform-tests for writable streams is not particularly comprehensive enough, for me to really want to trust shipping what we've implemented. I'd prefer us to have the additional pounding that piping will afford us, before I flip this on.

Until all the parts are working (i.e. writable streams and piping) it will not be a huge improvement to have writable streams. Also it's easier to workaround the absence of WritableStream than the absence of interoperability of streams with other APIs (e.g. fetch PUT requests which don't accept ReadableStream in any browser AFAIK).

As promised, here is an updated link to streaming Sintel using writable streams if they are present. This can start within a few seconds in Chrome, but in Firefox has to wait until the entire 120mb video is downloaded and decrypted (and it may crash if it doesn't fit in ram):

https://demo.peergos.net/#6MDZhRRPT4ugkJuUfcPPhf1US9u7FvRALmj42mJ6e3yDibnLtqfhchE6Frm6Lf/6MDZhRRPT4ugkJuUfcZdxu6JLKyrLBE36Kasxb4jix7An4dbeiekpDF6h2fDBM/G5KGmeQzGbaA16NGXRR6MtzkTw54YABd8wxaYxZKecth/5Pf7SvriCWAvPWM2ubDhCXbFasZFmPTZEyjHLKJKa26tYxCxN1R?open=true

(In reply to ianopolous from comment #6)

Ianopolous, did you find a workaround? When looking at your website, it looks like this isn't downloading the entire file in RAM.
Regarding WritableStream, this feature becomes more & more important by the years. Nowadays, virtually every files need to be encrypted for security, an operation made easier (& client-side) by WritableStream. I know it requires piping - I'm just saying it is an important feature, that should be supported in the future.

(In reply to Florian Ernst from comment #10)

As promised, here is an updated link to streaming Sintel using writable streams if they are present. This can start within a few seconds in Chrome, but in Firefox has to wait until the entire 120mb video is downloaded and decrypted (and it may crash if it doesn't fit in ram):

https://demo.peergos.net/#6MDZhRRPT4ugkJuUfcPPhf1US9u7FvRALmj42mJ6e3yDibnLtqfhchE6Frm6Lf/6MDZhRRPT4ugkJuUfcZdxu6JLKyrLBE36Kasxb4jix7An4dbeiekpDF6h2fDBM/G5KGmeQzGbaA16NGXRR6MtzkTw54YABd8wxaYxZKecth/5Pf7SvriCWAvPWM2ubDhCXbFasZFmPTZEyjHLKJKa26tYxCxN1R?open=true

(In reply to ianopolous from comment #6)

Ianopolous, did you find a workaround? When looking at your website, it looks like this isn't downloading the entire file in RAM.
Regarding WritableStream, this feature becomes more & more important by the years. Nowadays, virtually every files need to be encrypted for security, an operation made easier (& client-side) by WritableStream. I know it requires piping - I'm just saying it is an important feature, that should be supported in the future.

Sorry, we've since increased the max file size to load in ram, so that link works fine now (and that link was erroneously to the trailer which is only 15mb). Here is an updated one to the full 120mb version. If you click on the file and then click download, it will give a warning in Firefox, before proceeding to load the entire file into ram. In Chrome it can stream directly to a file, 5MiB at a time, using WritableStreams. There is no work around possible that we are aware of.

https://alpha.peergos.net/#{"secretLink":true%2c"link":"#6MDZhRRPT4ugkJuUfceM6bPnpQKEj5dB2NqLxD1RxFn3oA3CusXayN8RReauEh/6MDZhRRPT4ugkJuUfcRzRbPpFimcBNJx2N9TJDnL4W3ETYhwdsWdvgCkXkwipF/6M2nCSamXWTfbha5pYsyMPNSYxp5HjEF4KBPYbVQzsJU/5Pf7SvAwvcgFn1Wt4yVFDPKTu8wR8KMY2ncL5qZfQAjdsdk3ebD"}

It seems like Safari now supports WritableStream: https://trac.webkit.org/changeset/266172/webkit.

Duping this to implementation bug in the new DOM-based implementation in bug 1735664. I am going to carry over some of the blocked bugs to the streams enabling bug 1749547.