I was investigating a reftest failure from my patches in bug 1180295 and I discovered that when reftests are run on "Android 4.0 API 11+" the snapshots basically contain garbage after y=672. The garbage is usually what happens to be lying around in the graphics buffer so it may often be the "right garbage" but it is nonetheless garbage. As evidence, see the try pushes at [1] and [2]. Both of these have the reftest-sanity/corners-3.html test modified. This test just draws a single colored pixel in the same spot but of a different color in test and reference cases, and checks that the images are unequal. If the pixel is located at y=671 the test passes, at y=673 the test fails. In m-c the pixel is at y=599 so it passes, but my patches in but 1180295 end up changing the "y=672" threshold to "y=564" and that causes this test to fail [3] along with corners-4. Incidentally I also believe this garbage-drawing behaviour explains why we get so many intermittent fuzzies on Android 4.0, and it also explains some strange behaviour we've seen in the past (e.g. in bug 1169331 and many others) where there's just random blocks of pixels that are slightly off in the bottom part of the snapshot. I believe the root cause here is that Android 4.0 tests run on a pandaboard where the physical screen height is 780 pixels and leaves 672 pixels available for Fennec. The Android 2.3 and Android 4.3 tests appear to be running in an emulator (at least according to TH) and so are not subject to this problem. It might be possible to work around this problem by setting a CSS viewport of 1000x800 (the standard reftest dimensions) but I'd be just as happy with turning off these tests entirely. That's not really my call though. [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b55949ae0f89 [2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=5075faf0a14d [3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3ead1d37f62

See the following for some particularly egregious examples of issues caused by this bug: 1) Bug 1184356 comment 1: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=11734055 In both reftest failures here there's just a random block of pixels that's off. 2) Bug 1182292 comment 47: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=12515290 In all of these failures you can clearly see that the snapshot is chopped at y=672. There's lots more examples if you look around in the intermittent-failure bugs that are for reftests on Android 4.0.

Android 4.0 API11+ tests are being phased out. "Android 4.0 API11+ opt" reftests no longer run on any tree except for try, and then only when specifically requested (you kind-of did, perhaps unintentionally). We have been having trouble replacing "Android 4.0 API11+ debug" reftests, but I have faith those efforts will soon prevail and the whole Android 4.0 / pandaboard platform will be retired.

Thanks for the info. The R-1 failure from bug 1180295 was happening on both opt and debug, but if we're just retiring those tests I can disable them temporarily or something. Now that I understand why it's failing I don't have a problem with that, since it's only a couple of tests that are affected.

For now I'd like to disable this test on the pandas so that it doesn't interfere with my landing bug 1180295. Once Android 4.0 is running on an emulator with a bigger screen size we can re-enable it. (For those following along, gbrown pointed me to bug 1186615 as one of the main bugs for that work).

I'm starting to suspect that: (a) The Android 2.3 reftests also have this problem, except that the threshold is at y=816 instead of y=672. For Android 4.3 it would be 1280 (and therefore taller than the reftest area of 1000 pixels, so no problem there). (b) It's not actually "garbage" that gets drawn, it's just that nothing gets drawn. If there's any pixel differences in that area they're going to get missed.

Can we just increase the screen size on the emulator? Panda might be harder... I suppose part of the fix for bug 1156817 could fix this too

We can change the emulator screen size fairly easily; just tell me what you want. Does this relate to --ignore-window-size (which we use on all Android reftests)? http://hg.mozilla.org/mozilla-central/annotate/beb9cc29efb9/layout/tools/reftest/remotereftest.py#l505

I'd like to bump up the Android 2.3 emulator screen size to the same dimensions as the Android 4.3 emulator screen. If I can do that in a try push let me know, so that I can test it out (or feel free to do it if you have some spare cycles). Thanks!

(for the record, the try push at https://treeherder.mozilla.org/#/jobs?repo=try&revision=00fa24cafa6c, where I filled the canvas with pink before snapshotting the page, shows that the canvas is definitely not getting filled as it should be).

Existing settings from hardware-qemu.ini: Android 2.3 hw.lcd.width = 1024 hw.lcd.height = 816 Android 4.3 hw.lcd.width = 720 hw.lcd.height = 1280

1024x1280 would probably the best for both, although for now I only care about increasing that 816 to something to >= 1096.

This patch changes the screen dimensions for Android 2.3 to 1024x1280. https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4a97f87d0b0 shows it running without any other changes.

gbrown's patch takes care of the Android 2.3 issues I was seeing. There's still more issues on Android 4.0 (running on the pandas). A lot of the those tests are hidden on m-c but the debug reftests are still running, and a handful of those are failing with the dynamic toolbar patches (same as corners-3 and corners-4 for which I put up attachment 8647633 [details] [diff] [review]). I'll need to skip or random-if those as well. The relevant try push showing those failures is at https://treeherder.mozilla.org/#/jobs?repo=try&revision=19300669182d - see the R3 failures under "Android 4.0 API11+ debug".

Updated to also random-if the ones that fail intermittently. The set of tests is 100% consistent over all the try pushes I've done so far, although I'm not really sure why it's just those tests that are exhibiting this. Maybe there's some other test in R6 that's screwing things up.

Test manifests were updated in bug 1251013.