Closed Bug 1523390 Opened 6 years ago Closed 3 years ago

Flash NetStream connection requests don't/can't close which can cause "hanging"

Categories

(Core Graveyard :: Plug-ins, defect, P5)

defect

Tracking

(firefox-esr60 wontfix, firefox65 wontfix, firefox66 wontfix, firefox67 wontfix, firefox68 wontfix)

RESOLVED WONTFIX
Tracking Status
firefox-esr60 --- wontfix
firefox65 --- wontfix
firefox66 --- wontfix
firefox67 --- wontfix
firefox68 --- wontfix

People

(Reporter: jdenney, Unassigned)

References

Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

Steps to reproduce:

MacOS Version: 10.13.6
FireFox Version: 64.0.2
Flash Version 32.0.0.114

Using a Flash Player which reuses a NetStream connection to an external video file can cause video playing to "hang", sample code:

<-------------------->
// from https://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/net/NetStream.html
var MyVideo:Video = new Video();
addChild(MyVideo);

var MyNC:NetConnection = new NetConnection();
MyNC.connect(null);

var MyNS:NetStream = new NetStream(MyNC);
MyVideo.attachNetStream(MyNS);
MyNS.play("http://www.helpexamples.com/flash/video/clouds.flv");
<-------------------->

Subsequent calls to NyNS.play() with different video files (or the same one) will cause simultaneous http requests for the video files. There does not seem to be a way to close or stop them, which causes hanging after about 4-5 subsequent requests (with sufficiently large video files or slow connection speeds). In
comparison, Chrome (71.0.3578.98) will close previous requests and hanging does not occur.

100% Reproducible (also in FireFox safe mode).

I've tested out closing and destroying previous NetStream connections directly from within Flash with no success.

Actual results:

Video playback stops and "hangs" until previous requests complete (which can take 30 minutes or longer).

Expected results:

Video playback unaffected and no "hanging". Previous requests/connections can be stopped/closed to save bandwidth/resources.

Please provide steps to reproduce.
That means that you describe how I can reproduce the bug without knowing anything about internal flash things.

Flags: needinfo?(jdenney)

Hi, J Denney, I've referred this to the group working on Flash related issues. Since Flash is nearing end of life, I'm not sure of what priority can be given to it.

Component: Untriaged → Flash (Adobe)
Flags: needinfo?(jdenney) → needinfo?(cpeterson)
Product: Firefox → External Software Affecting Firefox
Version: 64 Branch → unspecified

J Denny, thanks for reporting this issue. Did this NetStream functionality work correctly in earlier versions of Firefox or Flash?

We did enable a new Flash sandbox on macOS in Firefox 62 (released in September 2018, bug 1474375) that caused some regressions, which were fixed in Firefox 65 (released on January 29, bug 1501121).

Can you still reproduce this issue in the latest Firefox 65 release?

Flags: needinfo?(cpeterson)

(In reply to Chris Peterson [:cpeterson] from comment #3)

J Denny, thanks for reporting this issue. Did this NetStream functionality work correctly in earlier versions of Firefox or Flash?

We did enable a new Flash sandbox on macOS in Firefox 62 (released in September 2018, bug 1474375) that caused some regressions, which were fixed in Firefox 65 (released on January 29, bug 1501121).

Bug 1501121 shouldn't be a factor here. It isn't related to the Mac Flash sandbox.

We do have one known issue with the Mac Flash sandbox that is fixed in 66 and so not available in Release yet. That is bug 1525625 which affects changing setting in the Flash settings panel.

Can you still reproduce this issue in the latest Firefox 65 release?

A better test would be to try with the Mac Flash sandbox disabled.

Setting dom.ipc.plugins.sandbox-level.flash=0 in about:config turns off the Mac Flash sandbox. If the problem is no longer reproducible with the about:config change, that's a good indicator the Flash sandbox is causing the problem.

J Denny, could you test with the about:config change? It disables the Mac Flash process sandbox which is a security protection which mainly limits where the Flash plugin can write to on your filesystem.

Will look into a way to reproduce this locally.

Flags: needinfo?(jdenney)

No problem. Thanks for the response! I will have to confirm next week about performance in previous versions, but considering our regression team just discovered the issue within the past few months, I would imagine that it did work without issue in the past.

I was still able to reproduce the issue with 65. Next week, I'll be able to test out older versions of Firefox. Feel free to suggest specific versions that might help isolate when the issue was introduced.

I will test out 66 ASAP and also 65 with the recommended config changes.

Unfortunately, I do not have the means to provide/compile a sample swf at the moment, but I can test myself with our existing swf. If you're able to compile the example in my original post, I do think it'd be possible to replicate just by making multiple play() calls of sufficiently large files, such that you can get 4-5 requests occurring simultaneously. The 5th-6th play call should "hang")

Again thanks!

Flags: needinfo?(jdenney)

Unfortunately 66 and 65 with sandbox disabled still show the symptoms. I have a bit more testing that I should be able to try out next week. The swf I tested does not explicitly attempt to close previous connections, so would like to test one that does next week.

However, the main reason why I reported the issue was the explicit difference in behavior between Chrome and Firefox. I thought it could be the root cause of other related issues. For some reason, in Chrome, previous requests are closed/halted when "play()" is subsequently called, allowing any number of additional requests to succeed. In Firefox, some resource limit is getting hit as previous requests stay open and additional requests past whatever limit must wait until previous requests complete. I saw a forum post saying Firefox has a limit of 6 concurrent requests to the same hostname, so perhaps that's the limit?

I'm not sure if the "fault" lies in Flash, the swf's, or the browser's implementation. However, Chrome's handling of this scenario is ideal in this case, for us, at least.

The priority flag is not set for this bug.
:marco, could you have a look please?

Flags: needinfo?(mcastelluccio)
Flags: needinfo?(mcastelluccio)
Priority: -- → P3

(In reply to J Denney from comment #6)

Unfortunately 66 and 65 with sandbox disabled still show the symptoms. I have a bit more testing that I should be able to try out next week. The swf I tested does not explicitly attempt to close previous connections, so would like to test one that does next week.

If you have a reproducible test case, the "mozregression" script can help you quickly isolate the exact code commit that caused the regression. mozregression automates the downloading and launching of old Firefox Nightly builds. You just need to run your test case and then click mozregression's "good" or "bad" button for each build. The whole process usually only takes 5-10 minutes.

https://mozilla.github.io/mozregression/

Unfortunately, I've not yet been able to prioritize running the mozregression script, but this is still on my radar and will do asap.

I ran mozregession (awesome tool btw!) and got the following output:

34:00.04 INFO: Narrowed nightly regression window from [2017-07-17, 2017-07-19] (2 days) to [2017-07-17, 2017-07-18] (1 days) (~0 steps left)
34:00.04 INFO: Got as far as we can go bisecting nightlies...
34:00.04 INFO: Last good revision: aff336ac161daa3ea350e59a288963edbd58ed39 (2017-07-17)
34:00.04 INFO: First bad revision: dece50457378ac4934afe9fb3c2a8054e8894588 (2017-07-18)
34:00.04 INFO: Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=aff336ac161daa3ea350e59a288963edbd58ed39&tochange=dece50457378ac4934afe9fb3c2a8054e8894588

It was also followed by many warnings such as:

34:51.26 CRITICAL: First build aff336ac161d is missing, but mozregression can't find a build before - so it is excluded, but it could contain the regression!
35:53.07 WARNING: Skipping build e0b0865639ce: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.e0b0865639cebc1b5afa0268a4b073fcdde0e69c.firefox.macosx64-opt'
35:53.09 INFO: There are no build artifacts on inbound for these changesets (they are probably too old).

So it looks like the issue has been around for a while and our manual regression team just recently caught it (most of the users for this service use Chrome). I'm glad I was at least able to verify that the issue did not previously exist. Let me know if I can be of more assistance. I'd be willing to bisect the commits if given a tool/instructions to do builds.

@ Jeromie, does the Flash Player currently call NPAPI's NPN_NewStream API? Twitter reports a problem with Flash NetStream connections hanging in Firefox. The problem appears to have started when Firefox removed the NPN_NewStream API (bug 1352559 way back in 2017), though Benjamin Smedberg asserted at the time that the Flash Player no longer used the API.

@ J Denny, thanks for bisecting this regression! It's a huge help. In this case, the regression range points to a very old build: Firefox 56 from 2017. That version predates Firefox's Flash sandbox for macOS (bug 1474375 in Firefox 62), so the problem would have been caused by something else. Looking at the regression range, bug 1352559 ("Remove NPN_NewStream") looks very suspicious considering this bug is about NetStream connections failing:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=aff336ac161daa3ea350e59a288963edbd58ed39&tochange=dece50457378ac4934afe9fb3c2a8054e8894588

I remember having someone audit the code (probably Matt) when working with Benjamin before that change was made, and that we concluded that the API wasn't used. It's a large codebase, so it's possible that we messed up, but it seems weird.

We definitely have some video stress tests where we play several FLVs simultaneously in our regression suite, but they're super old, and they're hosted on our LAN.

If someone has a URL that we can use to debug a reproducible case, I'd be more than happy to have someone take a look.

Flags: needinfo?(jeclark)

I don't have a test case, so I haven't actually reproduced the problem. J Denny's comment 0 says "Video playback stops and hangs until previous requests complete", which suggests that video playback does work. Without the NPN streaming API, perhaps Flash or Firefox falls back to downloading the full FLV file before proceeding to the next stream?

@ J Denny, do you have a public test case you can share that demonstrates the bug using the http://www.helpexamples.com/flash/video/clouds.flv stream from your example code in comment 0? I don't have a way to compile Flash ActionScript at this time.

I see other .flv files in http://www.helpexamples.com/flash/video/ but no player .swf file.

I'm in the same boat of not being able to compile Flash at the moment, but will work on that asap and try and provide a public test case. In the meantime, fyi, Flash Builder Premium is probably the fastest/easiest way to start compiling, it comes with a free month trial. However, for Mac, you will likely encounter a "Requires Java 6" error when running Flash builder. If so, the fix as of macOS 10.13.6, is to install "Java for macOS 2017-001" last seen available here: https://support.apple.com/kb/DL1572. After installing, Flash builder should be able to open successfully.

I've compiled an .swf that can be used for testing and replicating the issue, the file and it's source are available here: https://github.com/spacemunkay/ff_flash_stream_test.

The issue is easier to replicate by loading a large .flv file within the flash app, for convenience I've got one that's about 400MB that can be downloaded here https://www.dropbox.com/s/a91h9cu35b85fju/swing_dancing.flv?dl=0, but you'll need to find a place to host it, like S3, as it won't stream from dropbox.

I should note that I'm now working this on a volunteer basis, as the company resolution was to just just tell the small number of affected users to use Chrome. However, I still think it could be worthwhile to investigate in case this is a symptom of a larger issue.

Behavior in Chrome (74.0.3729.131):

  • Click "Load Stream" multiple times - can see that previous stream connections are closed before new connections are opened. Can open streams indefinitely without issue.
  • Click "Close Stream" - can see that stream connection is closed.

Behavior in Firefox (66.0.5):

  • Click "Load Stream" multiple times - can see that previous stream connections appear to remain open. After loading 5-10 streams, the Flash app becomes unresponsive and new stream connections don't play the video (at least until sufficient previous connections complete downloading).
  • Click "Close Stream" - can see that the stream connection appears to remain open and downloading continues.

Let me know if there's anything else I can provide!

Correction: I think "the Flash app becomes unresponsive" is incorrect. Actually, I think only video playback stops.

Thanks for the test case, J! It's a huge help.

Jim: I can reproduce the bug with J's test case. An engineer with NPAPI knowledge should check whether Flash or Firefox is responsible leaving the streams open.

I'm testing in Firefox 66.0.5 with Flash 32.0.0.192 and macOS 10.14.4. I haven't tested Windows.

I loaded RawStreamTest.html and swing_dancing.flv from web server running on my localhost (python -m SimpleHTTPServer and http://localhost:8000/swing_dancing.flv).

If I press the "Load Stream" button once, the stream starts playing after about one second in Firefox and Chrome. If I press the "Load Stream" button ten times, the stream starts playing after about one second in Chrome (as before) but after about ten seconds in Firefox.

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(jmathies)
Component: Flash (Adobe) → Plug-ins
Flags: needinfo?(jmathies)
Priority: P3 → P5
Product: External Software Affecting Firefox → Core
Resolving as wont fix, plugin support deprecated in Firefox 85.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.