Phil Ringnalda (:philor) Reporter
	Description • 12 years ago

Load https://secure.pub.build.mozilla.org/buildapi/running and sort by Submitted at. Just below the 5 zombies, you'll see a

release-mozilla-beta 44ad9c166d46 Android Tegra 250 release-mozilla-beta opt test jsreftest-1

which was submitted at 2013-01-23 09:35:05, with a Running since of somewhere within 10 minutes or so of when you loaded the page. Come back in 10 minutes or so, and you'll see it with a new Running since, because we've been infinitely retrying that job for a week now.

	Chris AtLee [:catlee]
	Comment 1 • 12 years ago

killed

	bhearsum@mozilla.com (:bhearsum)
	Comment 2 • 12 years ago

This is something happens with tegras sometimes AFAIK.

	Phil Ringnalda (:philor) Reporter
	Comment 3 • 12 years ago

Well, not really. If we set up a p0 infinite retry every single time we release a beta, then before long we'll be out of tegras. Apparently last week we only got one, but now from today we got five more.

	Phil Ringnalda (:philor) Reporter
	Comment 4 • 12 years ago

May well be two bugs if it's the same retry we've been suffering through everywhere that didn't get the Android signing patch, though, one for why release jobs are hitting it, and a separate non-Automation (Release) bug about turning the error message into something that doesn't retry, since right now just pushing to try from an old repo sets up every Android job infinitely retrying.

	Phil Ringnalda (:philor) Reporter
	Comment 5 • 12 years ago

Now we're down to just three (I presume thanks to tegras that disconnect before they get to the part that retries). One of the three is http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/None-candidates/buildNone/logs/release-mozilla-beta_tegra_android-armv6_test-jsreftest-1-bm19-tests1-tegra-build116.txt.gz, which is indeed the same symptom as the pre-signing-patch failures, "Remote Device Error: updateApp() call failed - exiting".

So we could treat the symptom by just making that failure not be a "Remote Device Error:" so that we don't RETRY from it, but I'm sure someone will want to look into why we're hitting it.

	Aki Sasaki (not active) Assignee
	Comment 6 • 12 years ago

I thought about this before, and we could test a fennec apk with any signature if we re-signed it with a testing key before testing.  (All tegras+pandas would need to have SUT agents signed with the same key, so unless the testing key == the nightly key this would be a very large reimaging party.)

We probably need to turn off sendchanges from release builds for the time being.  Afaik we never really looked at the results before releasing anyway.

	Aki Sasaki (not active) Assignee
	Comment 7 • 12 years ago

Ok, I was wrong about release signing preventing installation (Callek manually verified on a tegra).

Happy to be wrong.  Since this isn't true, bug 836861 should hopefully get us working.

	Aki Sasaki (not active) Assignee
	Comment 8 • 12 years ago

Turns out most tegras can't install release-signed builds.
My current theory:

Anything that has a nightly-signed apk installed will be unable to install a release-signed apk.  As tegras test more and more things (esp on try) the subset of tegras that can install release-signed apks shrink to near-zero.

Callek could have tested on a tegra that was able to install a release-signed apk in comment 7.

Given this, and given that we don't actually look at the release build tests, let's kill the release build sendchanges.

	Aki Sasaki (not active) Assignee
	Comment 9 • 12 years ago

This may be as simple as setting this line in our various fennec release configs to False:
http://hg.mozilla.org/build/buildbot-configs/file/5a2512247972/mozilla/release-fennec-mozilla-beta.py#l75

Doing some investigation.

	Aki Sasaki (not active) Assignee
	Comment 10 • 12 years ago

Running a dump_masters.sh to compare.  Pretty sure this is it, though.

	Aki Sasaki (not active) Assignee
	Comment 11 • 12 years ago

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

This also disables make package-tests.
Not sure if this is an issue.

	Justin Wood (:Callek)
	Comment 12 • 12 years ago

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

Review of attachment 710916 [details] [diff] [review]:
-----------------------------------------------------------------

sheep it

	Aki Sasaki (not active) Assignee
	Comment 13 • 12 years ago

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

http://hg.mozilla.org/build/buildbot-configs/rev/1da41328b1c9

	Aki Sasaki (not active) Assignee
	Comment 14 • 12 years ago

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

Transplanted: http://hg.mozilla.org/build/buildbot-configs/rev/b917827a70af

	Aki Sasaki (not active) Assignee
	Comment 15 • 12 years ago

This should be fixed as of the releases spinning today&tomorrow.