+++ This bug was initially created as a clone of Bug #1436058 +++ I'm filing this as a placeholder bug for the next webrender update. I may be running a cron script [1] that does try pushes with webrender update attempts, so that we can track build/test breakages introduced by webrender on a rolling basis. This bug will hold the try push links as well as dependencies filed for those breakages, so that we have a better idea going into the update of what needs fixing. I might abort the cron job because once things get too far out of sync it's hard to fully automate fixing all the breakages. When we are ready to actually land the update, we can rename this bug and use it for the update, and then file a new bug for the next "future update". [1] https://github.com/staktrace/moz-scripts/blob/master/try-latest-webrender.sh

The last update (bug 1436058) updated WR to cset 342bc314db94aa439b2001249c5f24ccfcbccc22. There are regressions from newer WR csets, detailed in bug 1436058, but in summary: - servo/webrender#2399 caused the number of windows reftest failures to spike dramatically, to the point where the log exceeds the max allowed size and turns the job red. - servo/webrender#2408 caused a couple of linux reftest failures, one of which is fuzzable but the other not.

The Linux reftest failure is fixed. Re-posting my comment from the last wr-update bug here: I can't reproduce reftest failures on Windows as we're seeing on CI - they mostly run fine on my local Win10 machine, but I have a possible theory on the Windows bustage. If I run the flexbox/ reftests on both Linux and Windows the memory usage is relatively stable. However, when I run the image/test/reftest reftests, there appears to be a significant memory leak. On Windows, the memory usage grows by ~100MB/sec until the process seems to go idle / timeout at ~4 GB allocated. Since the default Windows build is 32 bit, that would seem like an OOM is probably occurring somewhere. On Linux, I see the same memory growth, but since it's a 64-bit build, it doesn't have the same address space issues. Is it possible that this could be the cause of the Windows issues on CI? There are a couple of patches in this changeset that deal with managing lifetime of image and pipeline resources, could they be related?

(In reply to Glenn Watson [:gw] from comment #2) > However, when I run the image/test/reftest reftests, there appears to be a > significant memory leak. On Windows, the memory usage grows by ~100MB/sec > until the process seems to go idle / timeout at ~4 GB allocated. Since the > default Windows build is 32 bit, that would seem like an OOM is probably > occurring somewhere. On Linux, I see the same memory growth, but since it's > a 64-bit build, it doesn't have the same address space issues. > > Is it possible that this could be the cause of the Windows issues on CI? > There are a couple of patches in this changeset that deal with managing > lifetime of image and pipeline resources, could they be related? The Windows builds on the try pushes are all 64-bit, so this seems unlikely. Also the patches in the changeset that deal with lifetimes were in a number of the bisection try pushes that I did (bug 1436058 comment 17) and the problem only started occurring with PR 2399. It's possible that it's a combination of the two changes that causes the failures. Since I could reproduce at least one of the windows reftest issues on my windows machine I'll see if I can get a recording of it, maybe that will give us a clue as to what's going on.

WR @ eceed87325ef1dbbd220a4ccbb9e7860f2679428 with the R8 failure fuzzed https://treeherder.mozilla.org/#/jobs?repo=try&revision=1e055ce73a925afe1aca3c6bebb70f8863db03d4 https://treeherder.mozilla.org/#/jobs?repo=try&revision=84bac70750d9ccdc09b8e4df3dba0f6a16632e1a Linux R3 failure is still there (since this doesn't include the fix for that) as is the Windows redness WR @ ac8fb3f6c77eee57aeac67301b0e6ef9e0818c7a https://treeherder.mozilla.org/#/jobs?repo=try&revision=6cda8804c5fb0b7fec538f37f25a6726b31a9b72 https://treeherder.mozilla.org/#/jobs?repo=try&revision=677f5fafa5a8d167644ab395163b23cb9357d0b7 Down to just the windows redness

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #3) > Since I could reproduce at least one of the windows reftest issues on my > windows machine I'll see if I can get a recording of it, maybe that will > give us a clue as to what's going on. Quick update: I can consistently reproduce a problem with #2399 applied which doesn't occur without that patch. To reproduce, I build Firefox and run: MOZ_WEBRENDER=1 ./mach run layout/reftests/reftest-sanity/647192-1.html With #2399 the scrollbar area shows up as red, but without #2399 it renders fine. I attempted to get a recording using ENABLE_WR_RECORDING=1 but discovered that turning on recording made the bug go away. After some investigation it looks like that happens because turning on recording turns off external images. I filed bug 1437925 and servo/webrender#2411 to fix that. At least that revealed the issue is related to external images and #2399, if that helps any. I also tried to get a wr-capture of the bad rendering - I can get the capture, but the capture is not complete (missing files in externals/) so attempting to load it in wrench doesn't work. Trying to dig into that now.

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #5) > I can get > the capture, but the capture is not complete (missing files in externals/) I fixed this with bug 1437949. > so attempting to load it in wrench doesn't work. Trying to dig into that now. So now I have a capture, attached. Attempting to load it in wrench on my Windows machine doesn't work (the window seems to hang) but I can load it on the same wrench version (built from 65c68cc1d1) on my OS X machine. And it shows the red scrollbars. I'll make a capture of the good version as well, without #2399, and attach that for comparison purposes. Hopefully this will provide useful information to debug the problem.

Here's a capture of the good rendering. Unfortunately I can't get this capture to load on either windows (same problem as before) or OS X (I get a panic about "Unsupported image shader kind"). Note that the WR revision of this capture is different from the other capture.

WR @ ac8fb3f6c77eee57aeac67301b0e6ef9e0818c7a https://treeherder.mozilla.org/#/jobs?repo=try&revision=86417389def170a478482cd30b225b0b13dee95d https://treeherder.mozilla.org/#/jobs?repo=try&revision=9703ef870be184f9edac7d8f544612f2dc043297 Linux green, windows still backlogged

^ windows was red as expected WR @ 38df48c62c929a3603f3c6e6393c2be8f1c88903 https://treeherder.mozilla.org/#/jobs?repo=try&revision=064776f7e05b066c4ad40181ba397b751dcfd477 https://treeherder.mozilla.org/#/jobs?repo=try&revision=233f8acb03b49cc6f65cf29c350bec3a2bea1b8a Linux green, windows builds not starting yet (bug 1372172). This one should be green since it includes servo/webrender#2412 which was the fix.

Huzzah, the jobs finally finished and things are back to normal. WR @ d69bd28a055e2afb2ad234eea297dd45f9b4b4e5 https://treeherder.mozilla.org/#/jobs?repo=try&revision=51cf20720ce738e9a9bb1a1b65ec4a2efb797f69 https://treeherder.mozilla.org/#/jobs?repo=try&revision=dcf4de75ddde300949443de8daf27578871ebae4 Green

WR @ 377e2b22c61dd67ae4d746d50af09a7e5241bd9f https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9bfc8e4c45e341cb9132d5f8d41fbc9f0cd45ef https://treeherder.mozilla.org/#/jobs?repo=try&revision=2c203d0fb516d97996e6095d21432bcd6f62f8a3 Green

WR @ 4af31b8aa79d5a1f3c0242dea6be4876c71075c5 https://treeherder.mozilla.org/#/jobs?repo=try&revision=cbd465362bd27e87cd44be870fd773b7d57b49e5 https://treeherder.mozilla.org/#/jobs?repo=try&revision=a32fab72f6f62f3883b600b37fac282ab8b009c2 Green

WR @ 4af31b8aa79d5a1f3c0242dea6be4876c71075c5 https://treeherder.mozilla.org/#/jobs?repo=try&revision=af6cdfe20062628ca14af1f4f03be9231009fd4c https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6195da8596e7ba40b9fd639b776000ff25e976d Green

WR @ 4af31b8aa79d5a1f3c0242dea6be4876c71075c5 https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c6a0d0429e423f59a6ba85fa06e307a8ece6ead https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1e587a7a63748b191102772013a586a5d9fd82c Green

Patch to webrender_bindings/ for rayon update.

Patch to toolkit/ for rayon update.

WR @ 1a7bda5a12236f1ea727ae748d604e2e2a56b0bf https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd0828e3ac4e87764dd70d546559ed9ba53fb2a5 https://treeherder.mozilla.org/#/jobs?repo=try&revision=ef14411795fa9ac1741d3a2eb19e2a6e2648558e Bustage from rayon update in servo/webrender#2431 - glenn provided patches above which I'll apply now WR @ b711201fe2ae4ab1ee7a1dc4a055dc3d90c0ca05 https://treeherder.mozilla.org/#/jobs?repo=try&revision=4caf6a2d8282f5255bbee46ba622d0bb6083d907 https://treeherder.mozilla.org/#/jobs?repo=try&revision=69a21c213f1f538b7e90ef734b810ce6595d8ea7 Ditto

Comment on attachment 8951503 [details] [diff] [review] bindings.patch This patch will go into my queue for bug 1438892

Comment on attachment 8951652 [details] Bug 1437572 - Update reftest fuzziness from WR PR 2408. https://reviewboard.mozilla.org/r/220940/#review226876

Comment on attachment 8951651 [details] Bug 1437572 - Update webrender to 4af31b8aa79d5a1f3c0242dea6be4876c71075c5. https://reviewboard.mozilla.org/r/220938/#review226874

Pushed by kgupta@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/37d984fce90f Update webrender to 4af31b8aa79d5a1f3c0242dea6be4876c71075c5. r=jrmuizel https://hg.mozilla.org/integration/autoland/rev/751d00a65e68 Update reftest fuzziness from WR PR 2408. r=jrmuizel

https://hg.mozilla.org/mozilla-central/rev/37d984fce90f https://hg.mozilla.org/mozilla-central/rev/751d00a65e68