User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36 Steps to reproduce: I opened the following sites, each in a seperate tab, in Firefox 55.0b13: Facebook Twitter Hacker News vowe.net I then started a file based speed test, where I normally get the full 150 MBit/s I have. Actual results: The download speed is below 5MBit/s. As soon as I close the open tabs, download speeds instantly climbs back to normal. Expected results: Opened tabs shouldn't affect download speed.

Valentin told me on twitter that he is using this speed test: http://speedtest.tele2.net/ I will try to reproduce.

I tried to reproduce on both windows and mac, but I couldn't see a discernible difference. My ISP connection is only 50mbit, though, so I cannot test the full speed described in comment 0. Some notes about what I tried: * I was using the AMS-SPEEDTEST-1 server located in Amsterdam. * I used the 1GB download file over http. * The download file was served over http1.1 * I was logged in to facebook and twitter. * I ensured I was in the cohort with 4 processes like the reporter. Valentin, can you try to see if its a particular web site that is interacting with the download? What if you just close facebook? How much memory are your processes using? (You can check with about:memory.) How busy is the CPU while you are running the speed test? Does this reproduce on nightly for you? If so, you could try taking a profile in the slow and fast cases to compare: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler I'm not sure if the profiler gives us c++ symbols on beta builds. Markus, do you know if it does?

Download speed resumes to normal when I close the Facebook tab! When I reopen Facebook, speed drops again.

(In reply to Valentin from comment #3) > Download speed resumes to normal when I close the Facebook tab! When I > reopen Facebook, speed drops again. What if you open the tabs in comment 0 in a different order? Do you get the same result? I just want to rule out it being an issue with facebook running in the same content process. The other possibility is that something in your facebook feed is creating a lot of network traffic. If you open the network monitor in devtools on facebook do you see anything going on?

(In reply to Ben Kelly [:bkelly] from comment #2) > I'm not sure if the profiler gives us c++ symbols on beta builds. It does not.

(In reply to Ben Kelly [:bkelly] from comment #4) > (In reply to Valentin from comment #3) > > Download speed resumes to normal when I close the Facebook tab! When I > > reopen Facebook, speed drops again. > > What if you open the tabs in comment 0 in a different order? Do you get the > same result? I just want to rule out it being an issue with facebook > running in the same content process. > > The other possibility is that something in your facebook feed is creating a > lot of network traffic. If you open the network monitor in devtools on > facebook do you see anything going on? Reordering doesn't affect anything. As soon as I close or open Facebook, speed goes down/up massively. Devtools don't show any traffic after finishing loading Facebook.

(In reply to Valentin from comment #6) > Reordering doesn't affect anything. As soon as I close or open Facebook, > speed goes down/up massively. Devtools don't show any traffic after > finishing loading Facebook. Ok, can you see if it reproduces in nightly? https://www.mozilla.org/en-US/firefox/nightly/all/

(In reply to Ben Kelly [:bkelly] from comment #7) > (In reply to Valentin from comment #6) > > Reordering doesn't affect anything. As soon as I close or open Facebook, > > speed goes down/up massively. Devtools don't show any traffic after > > finishing loading Facebook. > > Ok, can you see if it reproduces in nightly? > > https://www.mozilla.org/en-US/firefox/nightly/all/ Happens in Nightly, too. Funny thing: I don't even need to close the tab, when I log out of Facebook, download speed resumes back to normal.

(In reply to Valentin from comment #8) > Happens in Nightly, too. Funny thing: I don't even need to close the tab, > when I log out of Facebook, download speed resumes back to normal. If you feel up to it, can you try getting a profile both in the slow and fast cases using the addon linked here? https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler Once you have the profile there is a "share" button that will give you a perma-link you can paste here.

(In reply to Ben Kelly [:bkelly] from comment #9) > (In reply to Valentin from comment #8) > > Happens in Nightly, too. Funny thing: I don't even need to close the tab, > > when I log out of Facebook, download speed resumes back to normal. > > If you feel up to it, can you try getting a profile both in the slow and > fast cases using the addon linked here? > > https://developer.mozilla.org/en-US/docs/Mozilla/Performance/ > Profiling_with_the_Built-in_Profiler > > Once you have the profile there is a "share" button that will give you a > perma-link you can paste here. Slow, Facebook logged-in: https://perfht.ml/2uW9Sjz Fast, Facebook logged-out: https://perfht.ml/2uWwK2f

I'm unable to see these at the moment, but Markus tells me they both show the threads are mostly idle. Valentin, if you hit ctrl-shift-e to open the network monitor on the facebook tab, do you see any traffic?

(In reply to Ben Kelly [:bkelly] from comment #11) > I'm unable to see these at the moment, but Markus tells me they both show > the threads are mostly idle. > > Valentin, if you hit ctrl-shift-e to open the network monitor on the > facebook tab, do you see any traffic? Nope, nothing. Really odd.

Patrick do you have any ideas how to further diagnose this?

The profiles in comment 10 do show some differences. In the "fast" profile we spend 10% of the time in OnInputStreamReady(). In the "slow" profile we spend 0.3% of the time in OnInputStreamReady().

if its not reproducible I might start looking at the profile, add-ons, even anti-virus and Firewall software.. it wouldn't be shocking to see any of those things hooking facebook..

(In reply to Ben Kelly [:bkelly] from comment #14) > The profiles in comment 10 do show some differences. > > In the "fast" profile we spend 10% of the time in OnInputStreamReady(). > > In the "slow" profile we spend 0.3% of the time in OnInputStreamReady(). seems like that is just because there's a lot more to do when data is transferring fast, right?

(In reply to Patrick McManus [:mcmanus] from comment #16) > seems like that is just because there's a lot more to do when data is > transferring fast, right? I agree.

(In reply to Patrick McManus [:mcmanus] from comment #15) > if its not reproducible I might start looking at the profile, add-ons, even > anti-virus and Firewall software.. it wouldn't be shocking to see any of > those things hooking facebook.. We already tried safe mode. Valentin, did you use a new profile when you tried nightly? (I should have said to do that, sorry!) Do you have any antivirus on this machine? I believe you said its a mac. (In reply to Patrick McManus [:mcmanus] from comment #16) > seems like that is just because there's a lot more to do when data is > transferring fast, right? Yea. I was just noting that there are some differences which match the problem.

(In reply to Ben Kelly [:bkelly] from comment #18) > (In reply to Patrick McManus [:mcmanus] from comment #15) > > if its not reproducible I might start looking at the profile, add-ons, even > > anti-virus and Firewall software.. it wouldn't be shocking to see any of > > those things hooking facebook.. > > We already tried safe mode. > > Valentin, did you use a new profile when you tried nightly? (I should have > said to do that, sorry!) > > Do you have any antivirus on this machine? I believe you said its a mac. > > (In reply to Patrick McManus [:mcmanus] from comment #16) > > seems like that is just because there's a lot more to do when data is > > transferring fast, right? > > Yea. I was just noting that there are some differences which match the > problem. I can reproduce it on 2 MacBooks (MacBook Air Mid 2012, MacBook Pro 2015). I tried via hotspot and LTE, no difference. I don't use any antivirus, nor do I have any FF add-ons installed. The profile is new, I resetted Firefox.

Valentin, can you check in other browsers? Chrome and safari?

(In reply to Ben Kelly [:bkelly] from comment #20) > Valentin, can you check in other browsers? Chrome and safari? No issues in Chrome and Safari. I also tried another Speedtest (http://speed.hetzner.de/), same problem. The problem does NOT happen in FF 54.

shot in the dark, can you go to about:config and set network.http.throttle.enable to false and try to reproduce?

I was starting to wonder if something like this is happening: 1. FB in Valentin's geo-region or particular user account setup is using a long polling XHR for something. 2. Our network stack sees the XHR as a high priority thing. 3. Deprioritizes the background download as a result. I have no idea if we try to prioritize like that, though.

(In reply to Patrick McManus [:mcmanus] from comment #22) > shot in the dark, can you go to about:config and set > network.http.throttle.enable to false and try to reproduce? Yes! This did the trick. As soon as I set it to false, speed is normal. Back to default, speed slow.

ok. honza will need to investigate further. I would not expect the speedtest to be treated like a download.

(In reply to Patrick McManus [:mcmanus] from comment #25) > ok. honza will need to investigate further. I would not expect the speedtest > to be treated like a download. It is a file based speedtest, where you download a 1 gig file. I can also download the Firefox dmg, which is also slow while I have Facebook open.

Valentin, thanks so far for providing profiles and general info. I'll ask you for one more thing - an HTTP log. Please see https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging for details. If you are going to use about:networking to start logging, please do so as the first think after you've started the browser. Then reproduce the problem, close the browser, and send me the log to my bugzilla email (it may contain private data you may not want to go public). Also include notes about what was the test you used so that I can look for the URL used for the actual test being slowed down. Thanks!

(suspecting a comet-request or websocket being open by logged-in FB active session, standing for an active transaction which makes downloads throttle)

(In reply to Honza Bambas (:mayhemer) from comment #27) > Valentin, thanks so far for providing profiles and general info. I'll ask > you for one more thing - an HTTP log. > > Please see > https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging for > details. > > If you are going to use about:networking to start logging, please do so as > the first think after you've started the browser. Then reproduce the > problem, close the browser, and send me the log to my bugzilla email (it may > contain private data you may not want to go public). > > Also include notes about what was the test you used so that I can look for > the URL used for the actual test being slowed down. > > Thanks! You have mail. I had two sites open: Facebook and speedtest.tele2.net

why is this a download (which I had pereceived to be things in the download manager)

(In reply to Patrick McManus [:mcmanus] from comment #30) > why is this a download (which I had pereceived to be things in the download > manager) The site is designed as a file download performance test. Go here: http://speedtest.tele2.net/ And click one of the files. It has some explanatory text as well.

Thanks! As I expected, there is an active (never gets a response) transaction to https://5-edge-chat.facebook.com/pull?... which is apparently a push or a comet request. I presume that the load indicator on the tab you have Facebook open is not spinning? If not, this in-progress request is marked as a background load. There are two options: - track LOAD_BACKGROUND channels on the parent process (needs some IPC'ing, since this can be set after opening a channel on the child) and exclude those from the "active" list [complicated and not a complete fix] - throttle only for a reasonably short period of time (based on the number of active transactions) [very easy to do with a general effect]

(In reply to Ben Kelly [:bkelly] from comment #31) > (In reply to Patrick McManus [:mcmanus] from comment #30) > > why is this a download (which I had pereceived to be things in the download > > manager) > > The site is designed as a file download performance test. Go here: > > http://speedtest.tele2.net/ > > And click one of the files. It has some explanatory text as well. thanks. that's odd - but at least it matches the feature!

(In reply to Honza Bambas (:mayhemer) from comment #32) > Thanks! As I expected, there is an active (never gets a response) > transaction to https://5-edge-chat.facebook.com/pull?... which is apparently > a push or a comet request. > > I presume that the load indicator on the tab you have Facebook open is not > spinning? If not, this in-progress request is marked as a background load. > > There are two options: > - track LOAD_BACKGROUND channels on the parent process (needs some IPC'ing, > since this can be set after opening a channel on the child) and exclude > those from the "active" list [complicated and not a complete fix] > - throttle only for a reasonably short period of time (based on the number > of active transactions) [very easy to do with a general effect] Site is fully loaded, no spinner.

[Tracking Requested - why for this release]: How critical is this? Do we think we can get this into FF55 release given the late date? Or is it worth flipping the throttle pref off in release until a fix can be shipped?

(In reply to Ben Kelly [:bkelly] from comment #35) > [Tracking Requested - why for this release]: > > How critical is this? Do we think we can get this into FF55 release given > the late date? Or is it worth flipping the throttle pref off in release > until a fix can be shipped? I would rather flip the pref for release. This is still evolving and I'm afraid the patch may not be easily upliftable to release. But I think I can have a Beta fix at least soon (tomorrow).

Glad I could help! Keep going, FF55 is a blast. So happy to see such a leap in performance.

(In reply to Honza Bambas (:mayhemer) from comment #36) > I would rather flip the pref for release. This is still evolving and I'm > afraid the patch may not be easily upliftable to release. But I think I can > have a Beta fix at least soon (tomorrow). Ok, I filed bug 1386759 to flip the pref. I'll leave it to you all on the network team to sort out which way you want to land things. Thanks!

(In reply to Ben Kelly [:bkelly] from comment #38) > (In reply to Honza Bambas (:mayhemer) from comment #36) > > I would rather flip the pref for release. This is still evolving and I'm > > afraid the patch may not be easily upliftable to release. But I think I can > > have a Beta fix at least soon (tomorrow). > > Ok, I filed bug 1386759 to flip the pref. I'll leave it to you all on the > network team to sort out which way you want to land things. Thanks! Thanks.

Probably needs some tuning of all cases when we want to prolong the window (on new unthrottled transactions and on recv of data on -certain- transactions under -certain- conditions). Patrick, no problem if you don't get to this.

Comment on attachment 8893039 [details] [diff] [review] wip1 (backup) Review of attachment 8893039 [details] [diff] [review]: ----------------------------------------------------------------- f+ assuming I'm reading that right.. it changes things to from "throttling is in effect when a 'focused' transaction is live" to "throttling is in effect for a few seconds after a focused transaction starts or receives some data", right?

Track 56+/57+ as the download speed is dropped when enabling network.http.throttle.enable.

(In reply to Patrick McManus [:mcmanus] from comment #41) > Comment on attachment 8893039 [details] [diff] [review] > wip1 (backup) > > Review of attachment 8893039 [details] [diff] [review]: > ----------------------------------------------------------------- > > f+ assuming I'm reading that right.. it changes things to from "throttling > is in effect when a 'focused' transaction is live" to "throttling is in > effect for a few seconds after a focused transaction starts or receives some > data", right? Yes, that's exactly what this aims at. Thanks.

- we throttle only for 3 seconds (arbitrarily chosen) from the last: - transaction activation - receive of data on an an unthrottled transaction for the active tab - the rest should be explained in code comments https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4292d4f99a37bf3e1c142994492f3ec649a2847

disabled in 55 by bug 1386759

better try? https://treeherder.mozilla.org/#/jobs?repo=try&revision=4031471f84965274750b839759b39da488092aed

Fixed typo.

Needs rebasing.

Pushed by ryanvm@gmail.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/eb96d03a53a2 Perform HTTP response throttling only for a limited time after new transactions are actived. r=mcmanus

Nevermind :)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #50) > Nevermind :) Deps on bug 1387090. Do you want me to express dependencies more visibly for you next time?

A note clarifying the landing order is always appreciated :)

What's the plan for 56 here? Are we going to try to backport fixes for this issue or go the disabling route?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #54) > What's the plan for 56 here? Are we going to try to backport fixes for this > issue or go the disabling route? Preferably, I'd like this to bake few days and then uplift to beta. But if we want a fix right now for 56, let's go the disable way as for 55 (bug 1386759)

Comment on attachment 8894496 [details] [diff] [review] v1.0.1 I'd like to have two beta cycles for this feature before 57 release. Throttling is one of the most important CDP project features, hence this needs a good tuneup. Approval Request Comment [Feature/Bug causing the regression]: bug 1312754 [User impact if declined]: under certain relatively common situations a major slowdown of running downloads [Is this code covered by automated tests?]: we don't have a test for this feature in particular (it's very hard to test in automation), but the code paths are well exercised by all our common tests [Has the fix been verified in Nightly?]: yes [Needs manual test from QE? If yes, steps to reproduce]: would be good, for STR see https://bugzilla.mozilla.org/show_bug.cgi?id=1386759#c18 [List of other uplifts needed for the feature/fix]: bug 1387090 [Is the change risky?]: this is untested, so it's hard to assess, in general the risk is low/medium [Why is the change risky/not risky?]: slightly risky, since it's not tested enough on m-c long enough [String changes made/needed]: none

(In reply to Honza Bambas (:mayhemer) from comment #56) > [Feature/Bug causing the regression]: bug 1312754 It's actually bug 1365307.

Comment on attachment 8894496 [details] [diff] [review] v1.0.1 More throttling tweaks, let's uplift this for 56 beta 2. I agree it's good for us to test this on beta before 57.

I tested this problem when I was working on bug 1386759 and I noticed that the problem only reproduced on mac OS 10.12 using Firefox beta 55.0b13, but it was fixed on Firefox 55.0 build 3. I retested the bug on Beta 56.0b2 using mac OS 10.12 and the bug is not quite reproducing. What I mean by that is that the internet speed is fluctuating from aprox. 3,5Mb/sec to 600Kb/sec and then back again to 3.5Mb/sec. The way I got those results was by navigating the Facebook page while the file was downloading. The mentioned results I got by opening the download doorhanger and checking the download speed from there. This behavior is not found on Chrome though. Any thoughts on this?

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #60) > I tested this problem when I was working on bug 1386759 and I noticed that > the problem only reproduced on mac OS 10.12 using Firefox beta 55.0b13, but > it was fixed on Firefox 55.0 build 3. > I retested the bug on Beta 56.0b2 using mac OS 10.12 and the bug is not > quite reproducing. What I mean by that is that the internet speed is > fluctuating from aprox. 3,5Mb/sec to 600Kb/sec and then back again to > 3.5Mb/sec. What is the expected maximum? If it is 3.5m/s then we do right. If it's expected to be more, then we have a bug, yes. > The way I got those results was by navigating the Facebook page while the > file was downloading. The mentioned results I got by opening the download > doorhanger and checking the download speed from there. This behavior is not > found on Chrome though. Any thoughts on this? We throttle downloads for short periods of time to give more bandwidth to loads of resources on the page you are navigating on. What you see is the expected behavior of that feature [0] but it might be too aggressive, yes. Important question is - do you perceive an improvement in page load times (while a download is running) when the throttling is on vs when it's off? The preference to switch it is "network.http.throttle.enable". If not, we may adjust the settings (see [1]) to be more aggressive in suppression during the active page load and more permissible to resume the download speed sooner after waters calm down. [0] bug 1386323 [1] https://dxr.mozilla.org/mozilla-central/rev/5322c03f4c8587fe526172d3f87160031faa6d75/modules/libpref/init/all.js#2171

(In reply to Honza Bambas (:mayhemer) from comment #61) > (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #60) > > I tested this problem when I was working on bug 1386759 and I noticed that > > the problem only reproduced on mac OS 10.12 using Firefox beta 55.0b13, but > > it was fixed on Firefox 55.0 build 3. > > I retested the bug on Beta 56.0b2 using mac OS 10.12 and the bug is not > > quite reproducing. What I mean by that is that the internet speed is > > fluctuating from aprox. 3,5Mb/sec to 600Kb/sec and then back again to > > 3.5Mb/sec. > > What is the expected maximum? If it is 3.5m/s then we do right. If it's > expected to be more, then we have a bug, yes. As far as I can tell 3,8 m/s is the maximum speed for downloading something (even on Chrome). > > The way I got those results was by navigating the Facebook page while the > > file was downloading. The mentioned results I got by opening the download > > doorhanger and checking the download speed from there. This behavior is not > > found on Chrome though. Any thoughts on this? > > We throttle downloads for short periods of time to give more bandwidth to > loads of resources on the page you are navigating on. What you see is the > expected behavior of that feature [0] but it might be too aggressive, yes. I can understand the part with giving bandwidth to load other content, but the difference in download speed is really big. That's my concern here. > Important question is - do you perceive an improvement in page load times > (while a download is running) when the throttling is on vs when it's off? > The preference to switch it is "network.http.throttle.enable". If not, we > may adjust the settings (see [1]) to be more aggressive in suppression > during the active page load and more permissible to resume the download > speed sooner after waters calm down. Yes, I can see an improvement in page load time, even when the download is running with throttle active. > [0] bug 1386323 > [1] > https://dxr.mozilla.org/mozilla-central/rev/ > 5322c03f4c8587fe526172d3f87160031faa6d75/modules/libpref/init/all.js#2171

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #62) > > What is the expected maximum? If it is 3.5m/s then we do right. If it's > > expected to be more, then we have a bug, yes. > > As far as I can tell 3,8 m/s is the maximum speed for downloading something > (even on Chrome). Se we are good. > > We throttle downloads for short periods of time to give more bandwidth to > > loads of resources on the page you are navigating on. What you see is the > > expected behavior of that feature [0] but it might be too aggressive, yes. > > I can understand the part with giving bandwidth to load other content, but > the difference in download speed is really big. That's my concern here. I don't think it's up to me or you to decide if this would be a showstopper for the throttling feature. If this is only a short-time drop in speed, I think we are good, but this is my opinion only. Note that people usually don't watch download speeds closely but that see the page loading progress. If you have time, you could experiment with following preferences: - network.http.throttle.resume-background-in - network.http.throttle.time-window These two can be considered a clamp for a minimum time (the "resume" one) and a maximum time (the "window" one) we engage throttling after the most recent network activity for the active tab. The resume-background-in pref will probably have an impact on resuming the download speed faster. The default is 1 second. You can experiment with lowering this value. But beside observing how this influences the download speed you must carefully observe what is the impact on the page load performance, whom improving is the main goal of throttling. Finding the right trade off will not be easy and I believe we can be a bit more aggressive here (give more BW to the page than to running downloads). I'll leave this testing up to you, it can be quite time consuming and from experience I don't expect a clear answer will come out of it, anyhow. Regarding verification of this bug I believe checking we resume the download speed in a reasonable time (few seconds) after a page load has finished is enough. > Yes, I can see an improvement in page load time, even when the download is > running with throttle active. This is the important thing! Thanks for confirming this.

(In reply to Honza Bambas (:mayhemer) from comment #63) > (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #62) > > > What is the expected maximum? If it is 3.5m/s then we do right. If it's > > > expected to be more, then we have a bug, yes. > > > > As far as I can tell 3,8 m/s is the maximum speed for downloading something > > (even on Chrome). > > Se we are good. > > > > We throttle downloads for short periods of time to give more bandwidth to > > > loads of resources on the page you are navigating on. What you see is the > > > expected behavior of that feature [0] but it might be too aggressive, yes. > > > > I can understand the part with giving bandwidth to load other content, but > > the difference in download speed is really big. That's my concern here. > > I don't think it's up to me or you to decide if this would be a showstopper > for the throttling feature. If this is only a short-time drop in speed, I > think we are good, but this is my opinion only. Note that people usually > don't watch download speeds closely but that see the page loading progress. > > If you have time, you could experiment with following preferences: > - network.http.throttle.resume-background-in > - network.http.throttle.time-window > > These two can be considered a clamp for a minimum time (the "resume" one) > and a maximum time (the "window" one) we engage throttling after the most > recent network activity for the active tab. > > The resume-background-in pref will probably have an impact on resuming the > download speed faster. The default is 1 second. You can experiment with > lowering this value. But beside observing how this influences the download > speed you must carefully observe what is the impact on the page load > performance, whom improving is the main goal of throttling. > > Finding the right trade off will not be easy and I believe we can be a bit > more aggressive here (give more BW to the page than to running downloads). > > I'll leave this testing up to you, it can be quite time consuming and from > experience I don't expect a clear answer will come out of it, anyhow. Thanks for the reply, I'll see when I have some time to spare to take a deeper look at this. > Regarding verification of this bug I believe checking we resume the download > speed in a reasonable time (few seconds) after a page load has finished is > enough. That was verified during my testing so I will go ahead and mark it verified fixed on 56.