Filed by: wkocher [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=115703474&repo=autoland https://queue.taskcluster.net/v1/task/PG5WDLvrRkOMqDtTMuTloQ/runs/0/artifacts/public/logs/live_backing.log Here's my attempt to track this down: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=5dd1a5ff2c0d41bfdb651cb66de48c93b8ea5c8b&noautoclassify&selectedJob=115703474&filter-searchStr=448e213e66eb9a06fe72b81389a37bb34e4dad7b&group_state=expanded&tochange=4741c5c80880c9825181a7f4e1fec0e139fcf4aa

Okay, this has instances before the merge from m-c to autoland this morning.

New range I'm testing: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=543c503f745c89360398fd3207ddaea5c262a807&noautoclassify&filter-searchStr=448e213e66eb9a06fe72b81389a37bb34e4dad7b&group_state=expanded&tochange=c5ea72577f794635a334a30780d104a1d89267a0&selectedJob=115717964

::mak, this has a lot of failures! I suspect many of these are retriggered jobs- could you look into this?

Here's my new range I'm testing, if it helps: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=1f7435a8c0d57fab04a0afb5aa0f0192b519b2f5&noautoclassify&filter-searchStr=448e213e66eb9a06fe72b81389a37bb34e4dad7b&group_state=expanded&tochange=9d89b33a35a3ae9e252a43038fa649180bf07491&selectedJob=115849195

Looking at the regression range from Wes, it seems it is definitely bug 1381049 that caused this. This will be my #1 job today - worst case we can possibly back it out, but I'd like to at least take a look first.

Comment on attachment 8888273 [details] Bug 1382444 - Stop places maintenance running during tests to avoid unnecessary overhead and intermittent issues. https://reviewboard.mozilla.org/r/159214/#review164604 Thank you! ::: testing/profiles/prefs_general.js:384 (Diff revision 1) > user_pref("marionette.prefs.recommended", false); > > // Disable Screenshots by default for now > user_pref("extensions.screenshots.system-disabled", true); > + > +user_pref("places.database.lastMaintenance", 7258114800); Please add a brief comment above explaining what this does and why.

After some investigations, we've determined that it appears to be the PlacesDBUtils.jsm changes from bug 1381049 that are causing this & a lot of other intermittents. The tests are writing a visit to the database using the read-write connection to the DB, and are then using a separate read-only connection to read the DB. However, if the write hasn't finished, then the read-only connection will get the previous snapshot. The changes in PlacesDBUtils.jsm are causing the writes to behave slightly differently, and what we are seeing is that the places maintenance task is sometimes kicking in and causing the writes to take much longer. This then causes the intermittent issues in the tests. Hence we've decided to turn off the maintenance task for tests - it doesn't need to run for them anyway.

BTW, try runs: With PlacesDBUtils.jsm backed out https://treeherder.mozilla.org/#/jobs?repo=try&revision=3e553649427dee3a84bcd7e7490613e3608dc0cc With the maintenance task disabled: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e8f1c0087e0d39e774f63d2dd8f8ae16b0eaa229&selectedJob=115910104

Pushed by mbanner@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1cfc47df313b Stop places maintenance running during tests to avoid unnecessary overhead and intermittent issues. r=mak

thanks for fixing this- should we consider this for talos/perf tests as well?

(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #16) > thanks for fixing this- should we consider this for talos/perf tests as well? It could make sense considered this code runs once a week in a normal profile. But maybe in general the idle service should not fire idle-daily on Talos and that may solve more issues than just one. Though, to do that, we'd need to set idle.lastDailyNotification to now (in seconds) and I don't know if that's feasible. Regardless, it's worth a separate bug for Talos.

good idea, I filed bug 1383896

this failure has come back, 25 instances in the last week: https://brasstacks.mozilla.com/orangefactor/index.html?display=Bug&bugid=1382444 all on linux64-debug. :standard8, do you think there is something easy to do here, or it would require a bit more investigation?

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #23) > this failure has come back, 25 instances in the last week: > https://brasstacks.mozilla.com/orangefactor/index. > html?display=Bug&bugid=1382444 > > all on linux64-debug. > > :standard8, do you think there is something easy to do here, or it would > require a bit more investigation? It looks like it was never quite 100% fixed. The interesting thing is I can't find any related bugs being marked as fixed from when it started ramping up again. That makes me think it is something else that could have affected timing or something else. Probably worth filing a new bug and we'll think about it there.