User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0

Actual results:

The scroll in Nighty and Developer is jerky.

Expected results:

The normal version the scroll is very smooth.

And the animations appears much smoother.

Does this happen in all websites?

Can you attach your about:support?

If the answer to the first question is yes, can you run mozregression to see what broke it?

Yes are in all.

PS: I cant use that software to compare the Nighty and Nomral version.

1. Open a window on youtube: https://www.youtube.com/
2. Create the second tab and open any video to run and move the tab to second monitor. (Both windws are Maximized)
3. Go to first window in Primry Screen and user the scroll.

In normal firefox are very smooth. In Nighty is like jumping, low fps.

So in Nightly you're using WebRender, and on release you're using d3d.

Moving to webrender as that's the most likely culprit.

Can you verify that going in nightly to about:config, then setting gfx.webrender.force-disabled to true, then restarting the browser, the issue is fixed?

Yes, its works. The scroll is like in the normal version, smooth.

But if i set that option true i get this bug in Nighty thoo: https://bugzilla.mozilla.org/show_bug.cgi?id=1576651

Hey can you get a capture of this in the profiler on nightly: https://profiler.firefox.com/

also if possible, can you go into the profiler settings and and make sure these threads are measured: GeckoMain, Compositor, WRSceneBuilder, WRRenderBackend, WRWorker, PaintThread

GeckoMain - Found
Compositor - Found
SceneBuilder - ?
RenderBackend - Found
Worker - ?
PaintThread - ?

https://drive.google.com/file/d/1a-ZZ53genfhWwwq1MG1VlYlAcJAh5gxz/view?usp=sharing

Hmm, odd, in my version of the profiler I can add additional threads by name. I'll ask Markus about this tomorrow.

premalink to a live version of this profile: https://perfht.ml/2PhyrFb

(to review later)

What resolution of screen are you using?

The resolution is: 1920x1080'

If i set 'gfx.webrender.all: true' to normal version the scroll is jerky too.
I this is because uses the WebRender, on DirectX 11 works fine.

If i play a video and i try to scroll, is much worse like with 75%.

If you turn on gfx.webrender.debug.profiler what fps do you get on each of the monitors?

You missunderstant the title, is not on two monitors only. With one monitor is the same. I just get the example of two because is much worse with two, and verry easy to see.

The fps from 'gfx.webrender.debug.profiler' is not shows the real fps when rendering. Its say above 40,50 every time but the scroll is like with 15fps every time.

I tried reproducing this locally on Intel 4600 and didn't see any problems. Are you able to capture the problem happening with a screen recorder and share the recording?

Also, can you install Intel Power Gadget from https://software.intel.com/en-us/articles/intel-power-gadget and report GT Frq0: and GPU Util%: with and without WebRender?

I record with my phone, 60fps, but the quality is low. I think the differance appears.
I put the csv in a .zip file. I can't upload both here so i put on google drive.

https://drive.google.com/open?id=1qrmR_a6163RpSuR7iz7eTOgmqQlcJ8wc

Does the performance improve if you disconnect one of your monitors?

No. Its the same with only the Primary.

Can you try getting a screen capture using https://obsproject.com/ with webrender.debug.profiler turned on?

Sure: https://drive.google.com/open?id=1vdeaquXZ-i6vEI7hFOicyzwYlY7Isq_8

OBS don't have screen capture. And the obs drops the frames. So, is not good.

Indeed. I tried OBS on my 4600 and it was unusable.

So new tactic: Can you share a profile from Nightly with the "Renderer" thread turned on?

How i make that? I don't know what you want to do.

• Make sure you have the profiler add-on from here https://perfht.ml installed in Nightly.
• Click on the "Gecko Profiler" icon on the toolbar.
• Expand the "Settings" section of the dialog that comes up
• In "Settings" there will be a "Threads:" section. Make sure the "Renderer" thread is clicked.
• Click the "Start" button to start recording.
• Scroll around to reproduce the problem.
• Use Ctrl+Shift+2 or click "Capture Profile" in the profiler dialog.
• Once the profile loads click the "Share" button in the top right.
• Paste the url that you get here.

https://perfht.ml/2O7GEJU

Great. This profile clearly shows that we're spending a bunch of time waiting in the Renderer thread. Unfortunately, it's not yet clear why.

The profile shows us spending 70% of the time in NtGdiDdDDIWaitForSynchronizationObjectFromCpu during texture upload.

Does the slowness only happen during video playback? Do you see similar slowdowns if you playback video on another site like vimeo? Do you see the same slowness if you set media.hardware-video-decoding.enabled to false?

1. No. The scroll is jerky on any page when scrolling. The video just increase problem / slowness / like lag spikes.
2. I think i response to 1.
3. Yes is the same or much worse.

Documenting something I found about the case with Jeff's help:

• all the time is spent in GPU cache uploads
• we had a very specific fix for this GPU - https://hg.mozilla.org/mozilla-central/rev/3064469c073d#l2.41
• GPU cache texture is the only thing this fix affects: it's the only RGBA32F texture that we are updating in parts, as opposed to re-uploading the whole thing
• with this fix, we are causing Angle to map a staging texture that is associated with the GPU cache texture, followed by a DMA/GPU copy from the staging into the actual resource...
• problem is that we shouldn't be mapping the same staging texture for writes while this copy is in flight, but this is what Angle does

Andronachi, can you try to get a gpuview capture of the scroll?

Here are some steps:

1. Download the Windows SDK from here: https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
2. Run the installer.
3. The only feature you need is called "Windows Performance Toolkit". Install that.
4. This will install gpuview at C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\gpuview
5. Inside this directory is README.txt that describes how to get a capture by running log.cmd
6. Upload the resulting Merged.etl file

Done: https://drive.google.com/open?id=173ixAW3tbACAypbpr-40VKg8wNKen2UK

About the driver @kvark i have: 20.19.15.4549, all drivers after this version is bugged, the vertical sync is strange.
Something like this: https://www.youtube.com/watch?v=TlRfgJsjLHU

Thanks. How much RAM do you have in that machine?

8GB - 256 Reserved to GPU.

I attached my Aida64 Report. If need my configuration.

Andronachi, could you try a workaround being tested in https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=267690358&revision=0e9ef27219cd714164c4893ebd0ebc45db574697 ?
The binary artifact path: https://queue.taskcluster.net/v1/task/P9avsJTeQLSlw8K7Vc7o5Q/runs/0/artifacts/public/build/target.zip

You'd need to enable WR in the prefs after unpacking it somewhere (set "gfx.webrender.all = true" in "about:config"), and restart the browser.
This version uses a new code path for GPU cache updates that may work better on your configuration.

I don't see much differance with WebRender, still persist.
I don't know how WebRender works with GPU but uses more CPU and GPU compared to DirectX.

Andronachi, can you also upload a gpuview trace with WebRender off in the same scenario?

Andronachi, thank you for testing that workaround! I have an idea on why it didn't work, but more importantly, I have another (different) take on that problem. If you have a chance, please try it out: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6925650c28eb6ae0b341e1af472c331d65727491
Artifact path: https://queue.taskcluster.net/v1/task/djCK7x8GRm-Y57yLis1NIw/runs/0/artifacts/public/build/target.zip

FYI. that previous take was to do the texture-texture copies when updating the GPU cache. It didn't help because we still need to upload the data to a staging texture, which was hitting the same slow path.
The new take is to enable the GPU scattering path for the cache update. It works by uploading the vertex data that is scattered using a pixel shader, so it doesn't involve direct texture updates at all.

Sure Jeff: https://drive.google.com/open?id=1u--nfb2q7vBzQLFxttGt6Vg_6cdLcl62

Dzmitry: Yes, i see a differance, better with i see like 70% maybe.

Question: Any idea why this page on DirectX is a little bit jerky after 11% and before 89% of the scroll ?

Andonachi, could you clarify what "better with i see like 70% maybe." means, please? Does it mean that you are 70% confident it's better, or that it's about 70% better, or something else?

Is better, but i don't know if is with 70%. Still have some jerky.

Adds a bit of code to detect haswell/broadwell on Intel and
switches GPU cache updates to use the scattering method.

I can maybe reproduce something like this on an i3 4140. I'm going to investigate further.

Hey Marian, isn't your second monitor actually a TV display? Looking at the recording from a different bug: https://www.youtube.com/watch?v=-uK2vy4zVIc

Actually, found this from the attached report:
Monitor Philips 226V (226V4) [21.5" LCD] (UK51348055187)
Monitor Toshiba TV (2325516843009)

So the second monitor is a TV display, somehow I believe the jerkiness is expected due to the average performance of intel graphic cards.
Jeff, could this be the cause? AFAIK, WebRender for intel was for 1920 x 1080 or smaller displays, don't believe TV displays belong there.

The reporter could reproduce the problem with only a single monitor attached.

Marian, can you please unplug the TV, restart your computer and try this out once more only on your main monitor?
Were all the profiles provided here done while both the main monitor and the TV were connected or only with the main monitor?

Just want to make sure this issue is not due the TV display. Really sorry for all the fussing around and thanks to Marian for being very cooperative.

@Timea Babos About the video image stuck with DirectX you can simly create that bug without a second monitor / TV.

1. Open the Firefox, make sure the window is Maximized.
2. Open a YouTube Home, Subscriptions (I don't think what the site is / metters)
3. Ceate a new tab and put the: https://www.youtube.com/watch?v=aT942ML7P98
4. Make sure the video is in 1080p 60fps (I think is because 60fps)
5. Drag the with the video tab down to make a new window and will be automatticaly maximized to monitor.
6. The video image is getting to be stuck, right click to taskbar, just random click outside the Firefox window.

Both have the same result no metter if the second monitor is plugged in or not.

The videos getting stuck is handled in the other submitted bug 1576651.
Are there any steps on a single monitor to reproduce the scroll jerkiness or its the same steps?

What Windows OS version do you have? The latest one that has several issues resolved is Version 1903 and OS build 18362.356

Pretty much the same steps.

You can just open the YouTube Subscriptions page and just scroll down. (The video just increase the jerkiness)

Andronachi, can you turn on gfx.webrender.debug.profiler and report the fps and renderer cpu time you get when you load https://jrmuizel.netlify.com/group-opacity-animated.html with WebRender on?

FPS: 11.09 - 9.43

The scroll is much smooth with 'gfx.webrender.picture-caching=false' but still sometimes jumping.
With the video is the same.

To confirm, you see better performance with gfx.webrender.picture-caching=false? When scrolling YouTube?
Do you see the same performance difference with gfx.webrender.picture-caching on and off problem on this page? https://webrender-bench.netlify.com/youtube.html

Yes. With that option 'false' is not that crazy and much smoother.
But still exists a litter bit of jerky.

I looked at the profiles (with JeffM) that force the regular texture upload path (as opposed to the workaround we have in Angle), and the closest suspect to jittered scrolling (which is still there) we could see was the Renderer time spikes. These spikes appear to be related to regular texture cache updates (not vertex textures, not GPU cache). Sometimes the driver decides to spend more time in UpdateSubresource.

In an ideal world where we could control the driver memory types, we'd be using some larger staging areas, and we'd have the predictable performance of copying and allocation. But doing this behind Angle is quite challenging. It's not clear at this point what the proper short term solution is.

No problem. Thanks all for time and trying.

WebRender will not be enabled on the 4600 for 70. We'll continue trying to figure out what's going on here.

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:kvark, could you have a look please?
For more information, please visit auto_nag documentation.

There is no update yet. The patch is not solving the problem, I'll mark it as such.

We landed something that might help with this in bug 1585404

Andronachi, has this improved in Nightly for you?

Still have dropped frames with WebRender :(

https://downloadcenter.intel.com/download/29313/Intel-Graphics-Driver-for-Windows-15-40-

Can you update to Intel 20.19.15.5107 driver, "Date: 1/10/2020" and see if there's any improvements? Your current 20.19.15.4549 driver is very old.

Sadly i can't do that GMA because all drivers after that version are bugged.
See: https://forums.intel.com/s/question/0D50P00004IoIY1SAN/i-have-a-bug-with-the-driver
And you can find more the same about graphic issues on: https://forums.intel.com/s/topic/0TO0P00000018NKWAY/graphics

Andronachi, we enabled gfx.webrender.compositor on Nightly on the 24th of January. Does it improve things for you?

Still have dropped frame. The performance of WebRender still very bad sadly.

Can you gather a new profile from Nightly?

Was fresh installed and the scroll still has dropped frames. Still have dropped frames if i make the window from maximized to 700x600 from 1920x1080.

Sorry. I mean a performance profile like in comment 24.

Done: https://perfht.ml/37hD4D1

Thanks. Do you have a sense for roughly when the frame dropping happened in the profile? Was there more near the end? Did it happen steadily throughout? Can you also make a profile with WebRender disabled?

With WebRender enabled is like: 90% of the time its happens
With DirectX 11 enabled is like: 0% i don't see any drop - https://perfht.ml/39pimTc

Both are on YouTube Home, scrolling down when the youtube stops loading more videos.

I need a better video card for WebRender insteated of the Intel HD 4600 ?
How i see in Task Manager, the WebRender uses more CPU and GPU that DirectX 11. Like *2

No we should be able to make WebRender work well on that card.

What GPU usage do you see in task manager with WebRender on vs off?

I make a video to see more detailed but its in 30 fps, so the dropped frames is not really on there.

WebRender: https://drive.google.com/open?id=12s5VrMf5yctVdlAIhwQ0D6xelZJmh8AU
DirectX11: https://drive.google.com/open?id=12hvWeNKGlpvGXc6Ig3bh4jL2ILHxp3uj

Sorry for the quality, focus on wrong place.

I can reproduce something like this on a 4600 and a 4400. I'll continue to investigate.

Looking at gpuview suggests that we're doing something bad with the texture cache.

More specifically when we reallocate the texture cache we do an allocation of 24MB this causes 20ms of time to be spend in MakeResidentCB

I discovered some interesting things about what's going on today. It looks like MakeResident operations block Map() calls. This means that when we resize the texture cache we get stuck blocking on the allocation/copy and don't continue submitting drawing commands until the copy is done. https://phabricator.services.mozilla.com/D63265 is an attempt at working around it.

A hacky patch for testing if this removes some more CPU sync points.

I took a look at this today.

As Jeff suggested, using scatter bus mode for the GPU cache texture helps a lot with synchronization blocks on CPU.

I also found that we're suffering from the same CPU sync points when updating the frame vertex textures (these are part of bind_frame_data). As a temporary hack, I changed that code to create/delete a fresh PBO each frame, rather than reusing the same PBO. This seemed to help a lot and removed most of the remaining CPU blocks showing up in the profile.

With that change, the scrolling seems mostly smooth to me. The GPU times mostly look reasonable - there is a spike every second or so while scrolling as new picture cache tiles are rasterized.

I attached a (hacky) patch to this bug which applies both those changes.

However, there are still occasional spikes I see in CPU sync points, during the vertex / GPU cache texture uploads. These happen rarely, but enough to cause a few spikes and frame drops. For example, the attached profile has the attached patch and you can see there is still a very significant amount of total overall CPU time spent inside WaitForSynchronizationObject.

I have some ideas for a better way to handle texture uploads that I've been wanting to try out for a while - I'm going to do some experiments with those tomorrow.

In the meantime, it would be interesting if anyone else is able to do some testing with the attached patch and see if that makes a noticeable difference on their machine too.

Bugbug thinks this bug is a regression, but please revert this change in case of error.

(In reply to Glenn Watson [:gw] from comment #84)

In the meantime, it would be interesting if anyone else is able to do some testing with the attached patch and see if that makes a noticeable difference on their machine too.

I still see jitter with your patch. The problem looks similar to what I see with scatter updates. No obvious CPU stalls, but occasional long GPU frames (10ms) which cause us to miss our budget

When I apply the hack patch above (that enables gpu scatter updates, and also rotates the vertex data textures), I can no longer see any obvious synchronization points in profiles.

When I scroll with that patch, what I generally see is very good frame times for the most part (~3-4 ms), and the occasional spike of ~10-12ms. This doesn't seem like it should cause much in the way of frame dropping, especially if running with DC enabled.

Those remaining large frame spikes I see are where every single tile seems to get invalidated and re-rasterized.

So, my current plan is:

• Investigate why all the tiles are getting invalidated and make a fix / temporary workaround for that.
• Create a build with that plus the sync fix patches above, and we can do some profiling on that to identify if there's any other frame dropping occurring.
• Work out a way to get the gpu scatter and vertex texture sync fixes landed, since I think they make sense anyway.
• Investigate why the "target init" / clear time is the major component of rasterizing each tile in this case - maybe we're doing something weird with retaining / discarding z-buffer or similar?

I found a bug where picture cache tiles were being evicted (and thus invalidated) incorrectly.

Fix for this is https://bugzilla.mozilla.org/show_bug.cgi?id=1627588. This improves the situation here slightly, although there is still at least one other case of seemingly unnecessary invalidation occurring (still investigating this).

Looking at my most recent trace that had Glenn's stuff but not the vsync the missed frames are not happening when we get a new scene. They seem to happen somewhat randomly.

Glenn's latest try for reference: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a6be48d1073a63e9207ca427772413ddb56f6cde

I got a new trace that includes the vsync work as well as Glenn's stuff. This shows us starting earlier which is good, but we still have really long frame submission times.

On one of the bad frames which is 3 frames after we get a new scene we spend:

• 7.2ms frame building compared to the typical 2.1ms
• 8.9ms rendering compared to the more typical 1.8ms
• During rendering we spend some time doing a texture allocation of a 768x256 R8 texture for use as a render target.
• There's also a texture free.

Took a new capture, this time with gfx.webrender.enable-gpu-markers=true. (Unfortunately GPUview seems to only give the start time of markers and not the end time)

In this capture we made most of our frame deadlines but we still have some frames where we're very close to missing.

Here are some example slow frames:

Frame 1:

• A 512x512x16 texture cache allocation during "texture cache update"
  • 2ms MakeResident in the paging queue
  • 1ms of sleeping on the cpu
  • overall texture cache update takes ~5ms?
• Creation of a 512x512x1 R8 RenderTarget
• Destruction of a 512x512x4 R8 RenderTarget

Frame 2:

• Happens during a scene build so there's some extra contention there and rayon has a garbage fire.
• Creation of a 256x256 R8 RenderTarget
• 2ms of texture cache update

I ran a new trace with the render target gc patch. I haven't looked super thoroughly yet but it looked like all of the slow frames that I saw were related to glyph cache texture upload.

I've seen this previously but we also seem to miss frames sometimes without anything obvious being slow which is weird.

This WIP patch contains some changes that may improve time in the renderer thread.

I still need to do further profiling and testing. It's difficult to accurately measure, so it would be good if Jeff could do some testing and see if it helps on that configuration, to see if we're on the right path. Even if these changes are useful, there are still frames that seem to take longer than expected and/or introduce occasional synchronization points, so further investigation is needed next week.

There are three parts to the patch. Each are orthogonal and it may be worth enabling / disabling each to see what the effect of each individual part are.

1. Inside ANGLE, use UpdateSubResource1 where available to supply DISCARD when updating an entire texture. This had quite a noticeable effect on my machine - reducing the time spent in WaitForSynchronizationObject inside D3D significantly.

2. Add a pooling system for standalone textures in the texture cache. This also had quite a noticeable effect on my machine. BUT, it doesn't currently GC / free any textures in the pool - so you'll run out of GPU memory at some point. It should be good enough for some testing of our test case though.

3. Change standalone textures to not allocate FBOs when the texture is created. I didn't notice any performance difference from this, but it's probably worth doing anyway for correctness / memory.

(In reply to Glenn Watson [:gw] from comment #95)

Created attachment 9141163 [details] [diff] [review]
optimization-wip.patch

This WIP patch contains some changes that may improve time in the renderer thread.

I did not notice any improvement from this patch.

(In reply to Jeff Muizelaar [:jrmuizel] from comment #96)

I did not notice any improvement from this patch.

I'm seeing some time in Map() with this patch which I don't think I saw without it. Maybe my working tree is missing some things...

Andronachi, a number of changes have landed that should improve this. Can you retest on the latest Nightly and see if things have improved for you?

Seams to be a big improve, i guess about 90%. There are not too big dropped frames like in FF74 that is use. Still have sometimes where the Dx11 not have one, but where Dx11 has some dropped frame the WebRender not.

Andronachi, that's good to hear. Can you try some other sites and if you notice dropped frames or other kinds of slowness can you post the urls here?

The https://en.wikipedia.org/wiki/Earth seams a little and is not much going on there.

Thanks. I've filed bug 1636616 to track that page specifically.

Andronachi, https://en.wikipedia.org/wiki/Earth should be improved in the latest Nightly? Do you still see the problem you were seeing there? Are there any other pages where you see problems?

https://www.w3schools.com/PHP/DEfaULT.asP seams to use too much CPU and GPU

Bug 1635610 should help with the w3schools issue.

(In reply to Andronachi Marian from comment #104)

https://www.w3schools.com/PHP/DEfaULT.asP seams to use too much CPU and GPU

This should be better now. Do you still see problems there? Any other problems anywhere else?

Seams to be much better now.

Not sure but there seams to be a little: https://www.amazon.com/AmazonBasics-Digital-Alarm-Clock-Nightlight/dp/B07DQWT15Y?pf_rd_r=9FJB7D75JTPVTQVBHWE9&pf_rd_p=277a7e11-1bef-478c-bd76-67176c3b2794&pd_rd_r=1ef8e1e5-095d-4521-ac08-c110a64bedf0&pd_rd_w=whDeA&pd_rd_wg=9XPZt&ref_=pd_gw_unk
On bottom seams to be more frequently.
In Dx11 is much worse on the CPU and GPU.

Hi. Can you check the https://www.maketecheasier.com/registry-values-for-group-policy-settings-windows/. Seams to have a bump on the start (One time scroll) and only with Ctrl+Shift+R is every time.

And this seams to be: https://addons.mozilla.org/ro/firefox/search/?platform=windows&q=Addon%20bar&page=2
With Ctrl+Shift+R is really bad.

https://addons.mozilla.org/ro/firefox/search/?platform=windows&q=Addon%20bar&page=2 - https://share.firefox.dev/3lx8fSO

https://www.qubittech.ai/ - WR https://share.firefox.dev/3nmsYJy - DX11 https://share.firefox.dev/3nmNbPo
This site works really well on Microsoft Edge 44.17763.831.0

Andronachi, that WR profile hides the problem because the renderer spends most of the time reading back the screenshot data. Could you take a profile with screenshots disabled, please?

I make only for qubittech: WR: https://share.firefox.dev/3f9iCtE Dx11: https://share.firefox.dev/2IBzDkx
And a video comparison: https://www.youtube.com/watch?v=h15wxgP9WOk

Found a new website on reddit that uses 100 GPU and is pretty bad for me too.
Only on WebRender is bad.

https://system76.com/

Closing this since the original issues appear to be fixed and it's getting a bit harder to follow. I opened bug 1728780 for the system76 website perf issue.