See: http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/116/testReport/%28root%29/test_homescreen_column_layout_py%20TestHomescreenLayout_test_homescreen_column_layout/test_homescreen_column_layout_py_TestHomescreenLayout_test_homescreen_column_layout/ Traceback (most recent call last): File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_client-0.13-py2.7.egg/marionette/marionette_test.py", line 296, in run testMethod() File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/tests/functional/homescreen/test_homescreen_column_layout.py", line 42, in test_homescreen_column_layout homescreen_settings.select_icon_layout('Four Columns') File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/apps/settings/regions/homescreen_settings.py", line 16, in select_icon_layout expected.element_present(*self._icon_layout_locator)) File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/wait.py", line 143, in until cause=last_exc) TimeoutException: TimeoutException: Timed out after 10.3 seconds I also could reproduce it locally. The problem is that the Settings app gets killed after getting back to the Homescreen app, which deletes the state it was in (in the Homescreen select part). Might be a regression similar as to bug 1172167, I don't know.

[Blocking Requested - why for this release]:

Just to be clear, this can be also reproduced manually on a Flame device, using 319MB.

Pull request to disable this test for the Flame device (until this gets fixed)

http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/homescreen/test_homescreen_column_layout.py#41 40 # No need to open homescreen setting again 41 # because it remembers where it left off And this is what's not happening anymore with the Flame at 319MB.

(In reply to Martijn Wargers [:mwargers] (QA) from comment #3) > Created attachment 8625070 [details] > https://github.com/mozilla-b2g/gaia/pull/30683 > > Pull request to disable this test for the Flame device (until this gets > fixed) Merged: https://github.com/mozilla-b2g/gaia/commit/3c01b0c7e4ac8e61806bf663f00e538382d1e0c9

Ok, I did a retest repeating this test 10 times. Out of the 11 times, it failed 6 times. It failed every time, the last 5 times. That leds me to believe something is growing in memory, while repeating the test (something in the Marionette framework?). Perhaps this test should be adjusted to not rely on where it left off after relaunch? Or is this supposed to always work for Flame devices with 319MB?

To me, this issue seems like a regression of the memory usage and then a bug needs to be filed with a memory report to highlight which app has seen its memory grow[1]. In the meantime, how much is this test intermittent? I suppose it fails starting from the 5th time because we don't reboot the phone (unlike on typical runs), so we can't say this is failing more than 50% of the time, can we? [1] https://wiki.mozilla.org/B2G/QA/Tips_And_Tricks#Getting_About_Memory_Logs_for_a_Bug

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #7) > To me, this issue seems like a regression of the memory usage and then a bug > needs to be filed with a memory report to highlight which app has seen its > memory grow[1]. > > In the meantime, how much is this test intermittent? I suppose it fails > starting from the 5th time because we don't reboot the phone (unlike on > typical runs), so we can't say this is failing more than 50% of the time, > can we? > > [1] > https://wiki.mozilla.org/B2G/QA/ > Tips_And_Tricks#Getting_About_Memory_Logs_for_a_Bug I think it's not happening often at all, but as you said, it depends on how many tests have run before. It seems that not rebooting the phone and rerunning tests is filling the phone's memory somehow. Perhaps we should just reboot the phone between tests? Anyway, perhaps this is a regression from bug 1094759 just like bug 1172167 is? I could do some testing with a repeat of 10 and see if there is a regression here.

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #7) > To me, this issue seems like a regression of the memory usage and then a bug > needs to be filed with a memory report to highlight which app has seen its > memory grow[1]. Ok, I filed bug 1177732 for this.

(In reply to Martijn Wargers [:mwargers] (QA) from comment #8) > Anyway, perhaps this is a regression from bug 1094759 just like bug 1172167 > is? > I could do some testing with a repeat of 10 and see if there is a regression > here. Ok, it seems like there is a different regression range here: Build ID 20150618010201 Gaia Revision b404c41c5471c31610e64defb74ec066b411e724 Gaia Date 2015-06-17 17:01:15 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/a3f280b6f8d5 Gecko Version 41.0a1 Working, but failing (4 times, but not at the end) often: Build ID 20150619010205 Gaia Revision a0df9c367a68764bdcf2e2e1c4d27f0d6ee165b8 Gaia Date 2015-06-18 18:49:14 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/2694ff2ace6a Gecko Version 41.0a1 Failing (6 times, although it had passed the 7th and 8th repeat): Build ID 20150619160206 Gaia Revision d8bac9577a537b5e6bafc3f7ed088af5ea60c99e Gaia Date 2015-06-19 13:21:50 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/c319f262ce3e Gecko Version 41.0a1 Failing the last 5 times: Build ID 20150620160206 Gaia Revision d8bac9577a537b5e6bafc3f7ed088af5ea60c99e Gaia Date 2015-06-19 13:21:50 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/a38c23ccca0a Gecko Version 41.0a1

Tentative regression range for gecko: https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2015-06-19+00%3A00%3A00&enddate=2015-06-19+23%3A21%3A50

Not blocking on low-mem issues.

Marking WONFTIX, sorry for the bug spam. If somebody still wants to work on this, please open a new bug.