Filed by: wkocher [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=84028589&repo=autoland https://queue.taskcluster.net/v1/task/LqFE572bRNKhJSB0xS9Ekg/runs/0/artifacts/public/logs/live_backing.log

wow, this test is failing very frequently as of late June 7th, primarily on linux debug in e10s mode. a recent log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=107467081 has this screenshot (not much value there): https://public-artifacts.taskcluster.net/D04ZGrLoQrmYaZDaqJEADg/0/public/test_info//mozilla-test-fail-screenshot_TFNwq6.png and this data in the log file: [task 2017-06-15T23:11:22.853043Z] 23:11:22 INFO - TEST-START | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html [task 2017-06-15T23:11:23.233023Z] 23:11:23 INFO - GECKO(3211) | ++DOMWINDOW == 30 (0x7f963ca71800) [pid = 3259] [serial = 124] [outer = 0x7f963e774000] [task 2017-06-15T23:11:24.082157Z] 23:11:24 INFO - GECKO(3211) | --DOCSHELL 0x7f963bb17000 == 2 [pid = 3259] [id = {6fd50d48-f464-432b-8b81-a593c323a615}] [task 2017-06-15T23:11:24.085284Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 29 (0x7f963c924800) [pid = 3259] [serial = 100] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?add&B&success] [task 2017-06-15T23:11:24.086825Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 28 (0x7f963bb18000) [pid = 3259] [serial = 99] [outer = (nil)] [url = http://example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?add&A&success] [task 2017-06-15T23:11:24.088313Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 27 (0x7f963c92f800) [pid = 3259] [serial = 101] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?add&B&success] [task 2017-06-15T23:11:24.089766Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 26 (0x7f963bb25800) [pid = 3259] [serial = 108] [outer = (nil)] [url = https://test2.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?clear] [task 2017-06-15T23:11:24.091069Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 25 (0x7f963ca67000) [pid = 3259] [serial = 112] [outer = (nil)] [url = about:blank] [task 2017-06-15T23:11:24.094663Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 24 (0x7f963c912800) [pid = 3259] [serial = 113] [outer = (nil)] [url = http://example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&A&success] [task 2017-06-15T23:11:24.103990Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 23 (0x7f963ca80800) [pid = 3259] [serial = 114] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&B&success] [task 2017-06-15T23:11:24.106137Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 22 (0x7f963d17d800) [pid = 3259] [serial = 115] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&B&success] [task 2017-06-15T23:11:24.108115Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 21 (0x7f963c92b800) [pid = 3259] [serial = 107] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?clear] [task 2017-06-15T23:11:24.111338Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 20 (0x7f963d180000) [pid = 3259] [serial = 104] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?remove&B&success] [task 2017-06-15T23:11:24.115173Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 19 (0x7f963d18b000) [pid = 3259] [serial = 105] [outer = (nil)] [url = https://test2.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?add&C&success] [task 2017-06-15T23:11:24.121648Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 18 (0x7f963e765800) [pid = 3259] [serial = 106] [outer = (nil)] [url = http://example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?clear] [task 2017-06-15T23:11:24.124344Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 17 (0x7f963d175800) [pid = 3259] [serial = 103] [outer = (nil)] [url = https://test2.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?add&C&failure] [task 2017-06-15T23:11:24.127838Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 16 (0x7f963c913800) [pid = 3259] [serial = 96] [outer = (nil)] [url = http://mochi.test:8888/tests/dom/tests/mochitest/localstorage/test_localStorageQuota.html] [task 2017-06-15T23:11:24.130874Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 15 (0x7f963bb30000) [pid = 3259] [serial = 109] [outer = (nil)] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html] [task 2017-06-15T23:11:24.132977Z] 23:11:24 INFO - GECKO(3211) | --DOMWINDOW == 14 (0x7f963ca74800) [pid = 3259] [serial = 102] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuota.html?add2&B&failure] [task 2017-06-15T23:11:24.462660Z] 23:11:24 INFO - GECKO(3211) | ++DOCSHELL 0x7f963bb15800 == 3 [pid = 3259] [id = {d12e604e-e3df-4ebd-bb5d-179cdf88e784}] [task 2017-06-15T23:11:24.465749Z] 23:11:24 INFO - GECKO(3211) | ++DOMWINDOW == 15 (0x7f963bb17000) [pid = 3259] [serial = 125] [outer = (nil)] [task 2017-06-15T23:11:24.503796Z] 23:11:24 INFO - GECKO(3211) | ++DOMWINDOW == 16 (0x7f963bb1e000) [pid = 3259] [serial = 126] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:24.969580Z] 23:11:24 INFO - GECKO(3211) | ++DOMWINDOW == 17 (0x7f963bb34800) [pid = 3259] [serial = 127] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:25.323880Z] 23:11:25 INFO - GECKO(3211) | ++DOMWINDOW == 18 (0x7f963c91c000) [pid = 3259] [serial = 128] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:25.587175Z] 23:11:25 INFO - TEST-INFO | started process screentopng [task 2017-06-15T23:11:28.619559Z] 23:11:28 INFO - TEST-INFO | screentopng: exit 0 [task 2017-06-15T23:11:28.717560Z] 23:11:28 INFO - Buffered messages logged at 23:11:25 [task 2017-06-15T23:11:28.721185Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | 500 bytes key A stored, expected 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 got 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 [task 2017-06-15T23:11:28.721587Z] 23:11:28 INFO - Buffered messages finished [task 2017-06-15T23:11:28.738294Z] 23:11:28 INFO - TEST-UNEXPECTED-FAIL | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | FAILURE: exception threw at http://example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&B&success: [task 2017-06-15T23:11:28.740251Z] 23:11:28 INFO - NS_ERROR_DOM_QUOTA_REACHED: Persistent storage maximum size reached [task 2017-06-15T23:11:28.741856Z] 23:11:28 INFO - onMessageReceived@dom/tests/mochitest/localstorage/interOriginTest.js:39:7 [task 2017-06-15T23:11:28.743288Z] 23:11:28 INFO - EventListener.handleEvent*@dom/tests/mochitest/localstorage/interOriginTest.js:9:1 [task 2017-06-15T23:11:28.744618Z] 23:11:28 INFO - GECKO(3211) | ++DOMWINDOW == 19 (0x7f963c92d000) [pid = 3259] [serial = 129] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:28.748817Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | 500 bytes key B stored, expected 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 got 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 [task 2017-06-15T23:11:28.755137Z] 23:11:28 INFO - GECKO(3211) | ++DOMWINDOW == 20 (0x7f963ca68000) [pid = 3259] [serial = 130] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:28.764472Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | Expected 2152924150 exception, expected 2152924150 got 2152924150 [task 2017-06-15T23:11:28.766200Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | Exception 2152924150 threw at http://example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add2&B&failure [task 2017-06-15T23:11:28.778860Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | Key B left unchanged, expected 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 got 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 [task 2017-06-15T23:11:28.782685Z] 23:11:28 INFO - GECKO(3211) | ++DOMWINDOW == 21 (0x7f963ca67000) [pid = 3259] [serial = 131] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:28.983561Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | Expected 2152924150 exception, expected 2152924150 got 2152924150 [task 2017-06-15T23:11:28.993986Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | Exception 2152924150 threw at https://example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&C&failure [task 2017-06-15T23:11:28.995463Z] 23:11:28 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | 500 bytes key C is NOT stored, expected null got null [task 2017-06-15T23:11:29.238837Z] 23:11:29 INFO - GECKO(3211) | ++DOMWINDOW == 22 (0x7f963bb20000) [pid = 3259] [serial = 132] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:29.320539Z] 23:11:29 INFO - GECKO(3211) | --DOMWINDOW == 21 (0x7f963ca6e800) [pid = 3259] [serial = 116] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add2&B&failure] [task 2017-06-15T23:11:29.323419Z] 23:11:29 INFO - GECKO(3211) | --DOMWINDOW == 20 (0x7f963d8ac000) [pid = 3259] [serial = 117] [outer = (nil)] [url = https://test2.example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&C&failure] [task 2017-06-15T23:11:29.326283Z] 23:11:29 INFO - GECKO(3211) | --DOMWINDOW == 19 (0x7f963ca64800) [pid = 3259] [serial = 118] [outer = (nil)] [url = http://test1.example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?remove&B&success] [task 2017-06-15T23:11:29.328867Z] 23:11:29 INFO - GECKO(3211) | --DOMWINDOW == 18 (0x7f963bb25000) [pid = 3259] [serial = 119] [outer = (nil)] [url = https://test2.example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&C&success] [task 2017-06-15T23:11:29.331076Z] 23:11:29 INFO - GECKO(3211) | --DOMWINDOW == 17 (0x7f963bb2e800) [pid = 3259] [serial = 120] [outer = (nil)] [url = http://example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?clear] [task 2017-06-15T23:11:29.449027Z] 23:11:29 INFO - TEST-PASS | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | Key A removed, expected null got null [task 2017-06-15T23:11:29.921105Z] 23:11:29 INFO - GECKO(3211) | ++DOMWINDOW == 18 (0x7f963bb32800) [pid = 3259] [serial = 133] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:30.099171Z] 23:11:30 INFO - Not taking screenshot here: see the one that was previously logged [task 2017-06-15T23:11:30.102317Z] 23:11:30 INFO - TEST-UNEXPECTED-FAIL | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | FAILURE: exception threw at https://example.com/tests/dom/tests/mochitest/localstorage/frameQuotaSessionOnly.html?add&C&success: [task 2017-06-15T23:11:30.103710Z] 23:11:30 INFO - NS_ERROR_DOM_QUOTA_REACHED: Persistent storage maximum size reached [task 2017-06-15T23:11:30.105033Z] 23:11:30 INFO - onMessageReceived@dom/tests/mochitest/localstorage/interOriginTest.js:39:7 [task 2017-06-15T23:11:30.106799Z] 23:11:30 INFO - EventListener.handleEvent*@dom/tests/mochitest/localstorage/interOriginTest.js:9:1 [task 2017-06-15T23:11:30.183941Z] 23:11:30 INFO - GECKO(3211) | ++DOMWINDOW == 19 (0x7f963ca6f800) [pid = 3259] [serial = 134] [outer = 0x7f963bb17000] [task 2017-06-15T23:11:30.365977Z] 23:11:30 INFO - GECKO(3211) | MEMORY STAT | vsize 1751MB | residentFast 99MB | heapAllocated 23MB [task 2017-06-15T23:11:30.565879Z] 23:11:30 INFO - TEST-OK | dom/tests/mochitest/localstorage/test_localStorageQuotaSessionOnly2.html | took 7714ms It really looks like this is the root cause: NS_ERROR_DOM_QUOTA_REACHED: Persistent storage maximum size reached :overholt, can you get someone from the dom team (this is the component I have, I assume there is a localstorage team maybe) to look at this asap? Given this high failure rate, I will check in on Monday afternoon and disable this bug if there is no active progress on this bug.

Jan probably knows what caused this or at least knows who to ask :)

luckily this has slowed down a little bit- 12 failures in the last 2 days- still frequent enough, but I am not itching to disable this test :)

still a steady failure rate, not enough to bug someone to investigate though.

This is a weird one, but I'm going to try and fix this as part of bug 1378716 because it seems like it's at least a little e10s related. Since both the test and its predecessor (*Only.html, no 2) seem to be run in the same process, my current best guess is something interesting may be happening with the group limits and races related to cleanup of the prior test. Overall test control flow structure seems sound enough by inspection, unless the state machine is being double-pumped somewhere.

Lots of failures here lately! OK to disable the test while you work on it?

Comment on attachment 8906140 [details] [diff] [review] skip test temporarily Review of attachment 8906140 [details] [diff] [review]: ----------------------------------------------------------------- Yes, fine to disable, sorry about the latency on this.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/f5c3628ecd2f Skip test_localStorageQuotaSessionOnly2.html for frequent intermittent failures; r=asuth