Description: Test is failing with high repro rate. Unsure what the fault is, as this does not reproduce manually. Test fails early on when attempting to reset the initial recorded, after performing in the FTU: File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 38, in test_cost_control_data_alert_mobile settings.reset_mobile_usage() Repro Steps: 1) Update phone to 20150624080416 2) Enable Data, disable Wi-Fi 3) Open the Cost Control app; enable data limit of 0.1 MB 4) Reset current observed data before performing 'test' Actual: Timeout occurs when attempting to reset data Expected: Data is reset and test continues Traceback (most recent call last): File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/.env/local/lib/python2.7/site-packages/marionette_client-0.13-py2.7.egg/marionette/marionette_test.py", line 296, in run testMethod() File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 38, in test_cost_control_data_alert_mobile settings.reset_mobile_usage() File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 105, in reset_mobile_usage self.tap_confirm_reset() File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 83, in tap_confirm_reset self.switch_to_settings_iframe() File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 42, in switch_to_settings_iframe *self._settings_title_locator)))) File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/.env/local/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/wait.py", line 143, in until cause=last_exc) TimeoutException: TimeoutException: Timed out after 20.1 seconds Environmental Variables: Device firmware (base) L1TC100118D0 Device firmware (date) 24 Jun 2015 12:38:44 Device firmware (incremental) eng.cltbld.20150624.153831 Device firmware (release) 4.4.2 Device identifier flame Gaia date 23 Jun 2015 18:48:14 Gaia revision eb0d4aefa62b Gecko build 20150624080416 Gecko revision 4cdc1a95a672 Gecko version 41.0a1 Reproducible manually: NO Repro frequency: 5/6

I'll take a look at this on Monday (data plans in a foreign country is probably very expensive).

Hrm, while running this test, I get this failure: TEST-UNEXPECTED-ERROR | test_cost_control_data_alert_mobile.py TestCostControlDataAlertMobile.test_cost_control_data_alert_mobile | TimeoutException: TimeoutException: Timed out after 40.0 seconds with message: Data usage bar did not breach limit Traceback (most recent call last): File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_client-0.16-py2.7.egg/marionette/marionette_test.py", line 296, in run testMethod() File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 66, in test_cost_control_data_alert_mobile message='Data usage bar did not breach limit') File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_driver-0.9-py2.7.egg/marionette_driver/wait.py", line 143, in until cause=last_exc) TEST-INFO took 224103ms SUMMARY ------- passed: 0 failed: 1 todo: 0 When I look at that code, I'm surprised it didn't fail on this before, actually.

Ok, when this failure is happening, there is an alert in the way that says you've reached the limit of mobile usage, just after the 0.1MB value is put in. Switching settings.toggle_data_alert_switch(True) and settings.reset_mobile_usage() makes it work. I guess there is something in the system that is already eating quite some band width, before setting the limit.

(In reply to Martijn Wargers [:mwargers] (QA) from comment #2) > When I look at that code, I'm surprised it didn't fail on this before, > actually. Never mind, I misunderstood the test a bit. But this test can be improved a bit, anyway.

This fixes the failure, but I wonder about this dialog I'm seeing while running this test shortly after putting in the 01.MB value. I'll attach a screenshot of that dialog shortly.

I'm seeing this dialog shortly after the 0.1MB is filled in. This is before the browser is opened even, so I don't know where this mobile data usage is coming from.

Ok, while running this test, I'm seeing mobile data usage already at 24.5MB or something before resetting mobile data. I wonder where that's coming from. I also wonder why this isn't deleted by a Gaiatest rerun. In any case, the pull request fixes the failure. I guess I should at least disable mobile data usage at tearDown.

Ok, I guess that somehow the Marionette communication is being added to the mobile data usage. Because I keep seeing numbers (MBs of data being used) that don't make sense. This looks like a regression and we should try to find out what caused it. I still think the pull request would be good to have, although it would stop catching issues like this. But I think this should better be covered in a regression unit test.

Seems to work in: Build ID 20150622010206 Gaia Revision d8bac9577a537b5e6bafc3f7ed088af5ea60c99e Gaia Date 2015-06-19 13:21:50 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/a38c23ccca0a Gecko Version 41.0a1 Fails in: Build ID 20150622160206 Gaia Revision 311c4e59936a407e64509f54fecb440d8a78e3c8 Gaia Date 2015-06-20 20:21:42 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/be81b8d6fae9 Gecko Version 41.0a1

https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2015-06-20+10%3A21%3A42&enddate=2015-06-22+23%3A21%3A42 I bet this is a regression from bug 1168658. Especially see bug 1168658, comment 18: " Fabrice, now that Bug 1157081 is fixed, it would be a good idea to flip the Safe Browsing prefs off by default (and test that TP still works) because Safe Browsing is way more data-hungry than TP. " I especially noticed that the longer you wait, the higher the mobile data usage, btw.

Johan, No-Jun. This is more a request for feedback. This would disable safebrowsing globally. As an alternative, I could disable these prefs in/as part of connect_to_cell_data(). Or I could only do it for the test that's failing in this bug.

Comment on attachment 8628348 [details] https://github.com/mozilla-b2g/gaia/pull/30781 Looks great! One small question left at the end, but I wouldn't block on it.

I've now added disable_cell_data() in tearDown to all the tests that turn mobile data on in their tests.

Comment on attachment 8628502 [details] [diff] [review] all.diff Review of attachment 8628502 [details] [diff] [review]: ----------------------------------------------------------------- Based on the discussion in bug 1168658, I think that's the best thing we can do.

Started Jenkins adhoc run with the last pull request (which also has the gaia_test.py changes in): http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/848/

It would be good to know what Dave Hunt thinks about this, disabling safe browsing globally (although it might be temporarily, until bug 1179770 gets fixed).

Comment on attachment 8628502 [details] [diff] [review] all.diff I think for testing purpose, this shouldn't be too much of a problem? (Unless some of our test cases rely on safe browsing and tracking protection, but I doubt it)

Comment on attachment 8628348 [details] https://github.com/mozilla-b2g/gaia/pull/30781 LGTM too.

(In reply to Martijn Wargers [:mwargers] (QA) from comment #15) > Started Jenkins adhoc run with the last pull request (which also has the > gaia_test.py changes in): > http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/848/ Ok, most tests pass and the failures are unrelated to this pull request. test_cost_control_data_alert_mobile.py is actually failing on Jenkins, but I think that is happening because bug 1177800 is checked in and Jenkins bitbar is using already a build with that checked in (and I didn't). I'll file a new bug about that and fix the locators. Merged: https://github.com/mozilla-b2g/gaia/commit/d22b4cd9d7a90fcd772bc9793fb0104eff019a41

(In reply to Martijn Wargers [:mwargers] (QA) from comment #19) > I'll file a new bug about that and fix the locators. I filed bug 1180062 for it.

I guess with bug 1180503 fixed, we could remove the safe browsing prefs (see all.diff diff). I filed bug 1181225 for it.