We need to deploy a new Watcher, to deploy the fix for bug 867360. There are concerns that updating the Watcher via sutagent fails, so this bug is about checking on that. (We deployed a new watcher.ini in bug 778177, but not a new Watcher...I'm not sure when we last updated the watcher, or tried to.)

I am trying to use an installApp.py-derived script to install a new Watcher via devicemanager.installApp() -> sut INST -> su -c "LD_LIBRARY_PATH=/vendor/lib:/system/lib pm install -r /mnt/sdcard/Watcher.1.16.apk". This works fine on my local phones, whether an old watcher is installed or not. But on the pandaboards, the pm install fails with INSTALL_PARSE_FAILED_INCONSISTENT_CERTIFICATES. We can get around this by first uninstalling, or possibly by changing how we sign the apk. If we uninstall, we lose the current watcher.ini, which is not included in the apk. Is that going to be okay? (Do we have a separate way of distributing the watcher.ini?)

(In reply to Geoff Brown [:gbrown] from comment #1) > (Do we have a separate way of > distributing the watcher.ini?) Yes - https://hg.mozilla.org/build/tools/file/c92f9eb6f601/sut_tools/verify.py#l232 So I am going ahead with the uninstall strategy.

This script uninstalls com.mozilla.watcher, pushes a watcher apk to devroot, installs that apk, and reboots the device (to trigger watcher restart). This does not include features likely required in production: - setting error flags - rebooting via relay, etc :jmaher - can you test on your tegra and panda?

Comment on attachment 756069 [details] simple script for installing watcher My general concerns here: * A one-off script for this stuff never scales well. We have had lots of one-off scripts in our code that have gotten left over and rewritten countless times. * Doesn't account for any way to check watcher is already updated (having a way to check the version and checking it was one blocker of a new watcher I had) * Newly imaged devices will not get an updated watcher, leaving us in an inconsistent state. We need to solve those before we deploy imo. Also what cert did we sign this with and why does it mismatch the cert we have in production?

this is a script to show how to update the watcher, not a script on how to roll this out in production. with that said I have not been able to update the watcher with this script on my tegra, it uninstalls and then disconnects (i.e. tegra appears to go offline or to sleep). I believe the watcher keeps it alive, we just need to figure out the right magical set of steps.

this works on my local tegra but fails on the panda board with a SIGSEGV. We could just roll out a new panda image with the updated watcher instead of updating on the fly. I am not sure if we can resolve this easily, but I am glad that there is a solution for the tegras!

(In reply to Joel Maher (:jmaher) from comment #6) > Created attachment 756711 [details] > my version of install watcher that works on tegras > > this works on my local tegra but fails on the panda board with a SIGSEGV. Thank you! I think this may work on pandas also, if we set LD_LIBRARY_PATH, like we do here: http://hg.mozilla.org/mozilla-central/annotate/7e3a4ebcf067/build/mobile/sutagent/android/DoCommand.java#l3306 ... or leverage that code.

Adding LD_LIBRARY_PATH doesn't crash the panda: status = dm._runCmds([{'cmd': 'exec su -c "export LD_LIBRARY_PATH=/vendor/lib:/system/lib; pm install /data/local/watcher.apk; am start -a android.intent.action.MAIN -n com.mozilla.watcher/.WatcherMain"'}]).split('\n') But it doesn't seem to bring the watcher back online after a reboot. There might be some other tricks to get it to start up.

Using ld_library_path for panda boards, I avoid the seg fault on installation. This is great. In fact this script returns and we are successful. The one caveat is the tegra needs to be powercycled, not just dm.reboot(). Callek, can you own taking this script and making it work with sut_tools?

some additional thoughts from Callek: * we need to update the tegra reimaging scripts to have the latest sutagent/watcher * we need to update the panda image with latest sutagent/watcher * how can we check the version of the watcher (right now it is in /data/data/com.mozilla.watcher/files/watcher.ini) I would like somebody to look at sutagent/devicemanager in detail and see if some of these commands I am executing could be done with existing commands...there are a lot of raw commands here. We should determine if we can allow for this in a test step (accept a reboot) or if this should be 100% manual.

(In reply to Joel Maher (:jmaher) from comment #11) > * how can we check the version of the watcher (right now it is in > /data/data/com.mozilla.watcher/files/watcher.ini) On service startup, Watcher writes its version string (now "Watcher Version 1.16") to /data/data/com.mozilla.watcher/files/version.txt. Is the version string also in watcher.ini?

watcher version 1.16 is the version. There shouldn't be a version in watcher.ini.

(In reply to Joel Maher (:jmaher) from comment #11) > I would like somebody to look at sutagent/devicemanager in detail and see if > some of these commands I am executing could be done with existing > commands...there are a lot of raw commands here. Comments: > dm._runCmds([{'cmd': 'exec su -c "dd if=%s of=/data/local/watcher.apk"' % target}]) > dm._runCmds([{'cmd': 'exec su -c "chmod 666 /data/local/watcher.apk"'}]) Use shellCheckOutput (with root=True) here. > time.sleep(5) Sort of unrelated but I'm not sure why this sleep should be necessary. uninstallApp should be blocking. > status = dm._runCmds([{'cmd': 'exec su -c "export LD_LIBRARY_PATH=/vendor/lib:/system/lib; pm install /data/local/watcher.apk; am start -a android.intent.action.MAIN -n com.mozilla.watcher/.WatcherMain"'}]).split('\n') Use installApp and startApplication (from droid.py) here. > status = dm._runCmds([{'cmd': 'exec su -c "pm install /data/local/watcher.apk; am start -a android.intent.action.MAIN -n com.mozilla.watcher/.WatcherMain"'}]).split('\n') Ditto

we are on an old version of mozdevice/devicemanager and we are not using droid.py. We do have shellCheckOutput available, let me try that.

I originally added the dm.reboot() to ensure the watcher was started. Now that the script has "am start", I think we can remove the reboot.

the tegras will reboot automatically as part of this process. I tried the shellCheckOutput and it produces a failure: Traceback (most recent call last): File "installWatcher.py", line 82, in <module> sys.exit(main(sys.argv)) File "installWatcher.py", line 68, in main if install(dm, devRoot, path_to_main_apk, "com.mozilla.watcher"): File "installWatcher.py", line 22, in install dm.shellCheckOutput("dd if=%s of=/data/local/watcher.apk" % target, root=True) File "/home/jmaher/mozilla/tools/sut_tools/mozdevice/devicemanager.py", line 81, in shellCheckOutput raise DMError("Non-zero return code for command: %s (output: '%s', retval: '%i')" % (cmd, output, retval)) mozdevice.devicemanager.DMError: Non-zero return code for command: dd if=/mnt/sdcard/tests/Watcher.1.16.apk of=/data/local/watcher.apk (output: 'sh: d: not found', retval: '127') It appears to be adding a space between every line in my command: "dd if=%s of=/data/local/watcher.apk" becomes: d d ' ' i f = / m n t / s d c a r d / t e s t s / W a t c h e r . 1 . 1 6 . a p k ' ' o f = / d a t a / l o c a l / w a t c h e r . a p k

(In reply to Joel Maher (:jmaher) from comment #17) > It appears to be adding a space between every line in my command: > "dd if=%s of=/data/local/watcher.apk" > > becomes: > d d ' ' i f = / m n t / s d c a r d / t e s t s / W a t c h e r . 1 . 1 6 . > a p k ' ' o f = / d a t a / l o c a l / w a t c h e r . a p k You need to specify the command like this: [ "dd", "if=%s" % foo, "of=/data/local/watcher.apk" ]

this uses the checkShellOutput! I also removed the sleep(5) as that isn't needed. Regarding launchApplication, that doesn't work. I had to bunch these together as the tegras need to have it launched before we lose connection to the device and they magically lose network connectivity shortly after installing the package. Because of that, bundling it up on the same command allows us to launch the watcher. The tegras require a power cycle due to the loss of network connectivity. The pandas do not require a power cycle/reboot, but we will not see the version file until the device is rebooted. This means we need reboot the device in order to verify installation.

(In reply to Joel Maher (:jmaher) from comment #19) > Created attachment 759324 [details] > update watcher for panda and tegra (1.1) > > this uses the checkShellOutput! I also removed the sleep(5) as that isn't > needed. Regarding launchApplication, that doesn't work. I had to bunch > these together as the tegras need to have it launched before we lose > connection to the device and they magically lose network connectivity > shortly after installing the package. Because of that, bundling it up on > the same command allows us to launch the watcher. Cool. To be clear, I don't consider using devicemanager methods to be a religion, especially for very specialized scripts such as these.

Is this scheduled to be deployed this week?

huh, looks like I never submitted my comment: Plan to deploy/test/etc. (in order): * manually update watcher on staging pandas and tegras, watch results for 1 day for any obvious problems * manually update tegra imaging process [imaging laptop] for new watcher and newest stable sutagent --* DCOps to be instructed on slightly changed procedure --* Instruct DCOps to image all staging tegras with new watcher/sutagent * someone [not callek] update panda image and deploy to mobile imaging --* be sure choice of naming reflects staging state --* image all staging (android) pandas with this new image * Watch newly imaged results for 1 to 2 days * Update 10% of production pools for each panda and tegra, manually, and at least 5% via DCOps imaging, documenting which devices were updated, and watch for 1-2 weeks * Update the rest of the pools and formally switch over imaging process to new image. The cautious approach here is due to the pain in performing a watcher downgrade. I expect we'll have this begun next week from the tegra side. I'll need someone to own updating the panda image though.

I manually tried to update the Watcher on both a tegra (tegra-018/foopy110) and a panda (panda-0307/foopy57). In both cases, the check of the version number (version.txt file) was not successful. I will attach the log files after posting this comment. I also updated the installation script to check both possible paths for the version.txt file: * /data/data/com.mozilla.watcher/files/version.txt (taken from jmaher's script https://bugzilla.mozilla.org/attachment.cgi?id=759324) * /data/app/com.mozilla.watcher/files/version.txt (taken from #ateam: http://logbot.glob.com.au/?a=search&c=mozilla%23ateam&q=version.txt&ss=8+Jun+2013&se=8+Jul+2013) I realise now that the mozdevice version being used was the one in the tools repository (http://mxr.mozilla.org/build/source/tools/sut_tools/mozdevice/) rather than the latest version from mozbase repository (https://github.com/mozilla/mozbase/tree/master/mozdevice/mozdevice) - maybe this is a reason it did not write the version.txt file? I will retest now, using latest mozdevice version, to see if this makes a difference. I'll replace the log files from panda-0307 and tegra-018 installations when this is done too. Please also note, I have not found a way to inspect what the current version of the watcher is, if it is older than when the version.txt file was introduced. However, in these two tests, also the version.txt file did not get created, so I do not know at the moment if it was successful upgrading but did not create the version.txt file, or it said it was successful, but was not...

It might be helpful if you could get the logcat showing the watcher startup. It should, for instance, log "writing version string to: <path>".

Hi Geoff, I've attached logcat output for both devices. Thanks!

let me run that again, with a higher logging level...

Relevant lines I see in panda logcat: 07-10 17:25:27.687 I/Watcher ( 1795): Watcher Version 1.16 ... ... 07-10 17:25:27.781 I/Watcher ( 1795): writing version string to: /data/data/com.mozilla.watcher/files/version.txt ... ... 07-10 17:28:25.750 I/SUTAgentAndroid( 1865): 10.12.130.21 : pull /data/data/com.mozilla.watcher/files/version.txt (i believe the last line is when the watcher version.txt is queried by jmaher's attached script). So the logcat looks successful, but for some reason the version.txt file seems not to be updated. Very strange.

Issue resolved! dm.getFile() does not work here because the file is rw for watcher app only... $>exec su -c "ls -l /data/data/com.mozilla.watcher/files/version.txt" -rw------- app_43 app_43 21 2013-07-10 17:25 version.txt return code [0] $> Two ways to get around this: 1) use 'exec su -c "cat /data/data/com.mozilla.watcher/files/version.txt"' (since root can read the file) 2) chmod the file to be readable by others, and use dm.getFile() as before After discussion in #ateam we decided to go with option 1, since this still works even if permissions are e.g. reset by some mechanism. I'll update the script, and re-attach it. Plan is tomorrow to roll out to the rest of staging (Thu 11 Jul 2013)...

An updated version of install_watcher.py, with the above fix...

Comment on attachment 780379 [details] install_watcher.py this looks fine. I am not sure why you need to: import site site.addsitedir(os.path.join(os.path.dirname(os.path.realpath(__file__)), "../lib/python")) maybe that is something common in newer releng scripts.

Yes, Joel, you are quite right, I will get rid of that - my mistake. It is not needed.

So I have now upgraded the watcher on all staging devices, which means all devices on the following foopies: foopy53, foopy54, foopy55, foopy56, foopy57, foopy58, foopy59, foopy60, foopy61, foopy62, foopy63, foopy64, foopy105, foopy106, foopy107, foopy109, foopy110 I am now going to run some sendchanges for the Androids (against https://tbpl.mozilla.org/?rev=05d3797276d3) to do a full test after the upgrade. I'm also updating our https://mxr.mozilla.org/build/source/tools/buildfarm/maintenance/foopy_fabric.py to include a method for displaying the watcher version. If all the tests are successful, this will be an indication we can move to the next stage, to get the imaging process updated, and get this rolled out to production.

OK, tests are running overnight - the staging buildbot test master for the mobile devices is: http://dev-master01.build.scl1.mozilla.com:8036/ (if anyone is curious to watch the progress).

(In reply to Pete Moore [:pete][:pmoore] from comment #39) > OK, tests are running overnight - the staging buildbot test master for the > mobile devices is: http://dev-master01.build.scl1.mozilla.com:8036/ (if > anyone is curious to watch the progress). pmoore - How did this turn out? Are we still moving ahead?

(In reply to Geoff Brown [:gbrown] from comment #40) > pmoore - How did this turn out? Are we still moving ahead? pmoore - ping? any update?

Hi guys, Sorry for the delayed update. I am having some difficulties, which I will now present here: When trying to compare results in staging vs. production, I am running the following sendchanges: buildbot sendchange --master localhost:9036 --username sendchange --branch mozilla-central-android-x86-talos --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-x86/1375317165/fennec-25.0a1.en-US.android-i386.apk buildbot sendchange --master localhost:9036 --username sendchange-unittest --branch mozilla-central-android-x86-opt-unittest --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-x86/1375317165/fennec-25.0a1.en-US.android-i386.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-x86/1375317165/fennec-25.0a1.en-US.android-i386.tests.zip buildbot sendchange --master localhost:9036 --username sendchange --branch mozilla-central-android-armv6-talos --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-armv6/1375317165/fennec-25.0a1.en-US.android-arm-armv6.apk buildbot sendchange --master localhost:9036 --username sendchange-unittest --branch mozilla-central-android-armv6-opt-unittest --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-armv6/1375317165/fennec-25.0a1.en-US.android-arm-armv6.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-armv6/1375317165/fennec-25.0a1.en-US.android-arm-armv6.tests.zip buildbot sendchange --master localhost:9036 --username sendchange --branch mozilla-central-android-noion-talos --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-noion/1375317165/fennec-25.0a1.en-US.android-arm.apk buildbot sendchange --master localhost:9036 --username sendchange-unittest --branch mozilla-central-android-noion-opt-unittest --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-noion/1375317165/fennec-25.0a1.en-US.android-arm.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-noion/1375317165/fennec-25.0a1.en-US.android-arm.tests.zip buildbot sendchange --master localhost:9036 --username sendchange --branch mozilla-central-android-talos --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1375317165/fennec-25.0a1.en-US.android-arm.apk buildbot sendchange --master localhost:9036 --username sendchange-unittest --branch mozilla-central-android-opt-unittest --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1375317165/fennec-25.0a1.en-US.android-arm.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1375317165/fennec-25.0a1.en-US.android-arm.tests.zip buildbot sendchange --master localhost:9036 --username sendchange-unittest --branch mozilla-central-android-debug-unittest --revision 05d3797276d3 --comments 'Merge inbound to m-c' --property buildid:20130731173245 --property pgo_build:False --property builduid:a39345585978443dbf392881eeab986e http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-debug/1375317165/fennec-25.0a1.en-US.android-arm.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-debug/1375317165/fennec-25.0a1.en-US.android-arm.tests.zip These sendchanges were scraped from the following 5 production logs, and adapted to run against Staging: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-x86/1375317165/mozilla-central-android-x86-bm61-build1-build76.txt.gz http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-armv6/1375317165/mozilla-central-android-armv6-bm64-build1-build41.txt.gz http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-noion/1375317165/mozilla-central-android-noion-bm61-build1-build61.txt.gz http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1375317165/mozilla-central-android-bm64-build1-build38.txt.gz http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-debug/1375317165/mozilla-central-android-debug-bm58-build1-build39.txt.gz The full list of builders that I know about are: Android 4.0 Panda mozilla-central opt test crashtest Android 4.0 Panda mozilla-central opt test jsreftest-1 Android 4.0 Panda mozilla-central opt test jsreftest-2 Android 4.0 Panda mozilla-central opt test jsreftest-3 Android 4.0 Panda mozilla-central opt test mochitest-1 Android 4.0 Panda mozilla-central opt test mochitest-2 Android 4.0 Panda mozilla-central opt test mochitest-3 Android 4.0 Panda mozilla-central opt test mochitest-4 Android 4.0 Panda mozilla-central opt test mochitest-5 Android 4.0 Panda mozilla-central opt test mochitest-6 Android 4.0 Panda mozilla-central opt test mochitest-7 Android 4.0 Panda mozilla-central opt test mochitest-8 Android 4.0 Panda mozilla-central opt test mochitest-gl Android 4.0 Panda mozilla-central opt test robocop-1 Android 4.0 Panda mozilla-central opt test robocop-2 Android 4.0 Panda mozilla-central talos remote-tcanvasmark Android 4.0 Panda mozilla-central talos remote-tp4m_nochrome Android 4.0 Panda mozilla-central talos remote-trobocheck Android 4.0 Panda mozilla-central talos remote-trobocheck2 Android 4.0 Panda mozilla-central talos remote-trobopan Android 4.0 Panda mozilla-central talos remote-troboprovider Android 4.0 Panda mozilla-central talos remote-ts Android 4.0 Panda mozilla-central talos remote-tsvg Android 4.0 Panda mozilla-central talos remote-tsvgx Android Armv6 Tegra 250 mozilla-central opt test crashtest Android Armv6 Tegra 250 mozilla-central opt test jsreftest-1 Android Armv6 Tegra 250 mozilla-central opt test jsreftest-2 Android Armv6 Tegra 250 mozilla-central opt test jsreftest-3 Android Armv6 Tegra 250 mozilla-central opt test mochitest-1 Android Armv6 Tegra 250 mozilla-central opt test mochitest-2 Android Armv6 Tegra 250 mozilla-central opt test mochitest-3 Android Armv6 Tegra 250 mozilla-central opt test mochitest-4 Android Armv6 Tegra 250 mozilla-central opt test mochitest-5 Android Armv6 Tegra 250 mozilla-central opt test mochitest-6 Android Armv6 Tegra 250 mozilla-central opt test mochitest-7 Android Armv6 Tegra 250 mozilla-central opt test mochitest-8 Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-1 Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-2 Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-3 Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-4 Android Armv6 Tegra 250 mozilla-central opt test robocop-1 Android Armv6 Tegra 250 mozilla-central opt test robocop-2 Android Armv6 Tegra 250 mozilla-central opt test xpcshell Android Armv6 mozilla-central build Android Armv6 mozilla-central nightly Android Debug mozilla-central build Android Tegra 250 mozilla-central opt test crashtest Android Tegra 250 mozilla-central opt test jsreftest-1 Android Tegra 250 mozilla-central opt test jsreftest-2 Android Tegra 250 mozilla-central opt test jsreftest-3 Android Tegra 250 mozilla-central opt test mochitest-1 Android Tegra 250 mozilla-central opt test mochitest-2 Android Tegra 250 mozilla-central opt test mochitest-3 Android Tegra 250 mozilla-central opt test mochitest-4 Android Tegra 250 mozilla-central opt test mochitest-5 Android Tegra 250 mozilla-central opt test mochitest-6 Android Tegra 250 mozilla-central opt test mochitest-7 Android Tegra 250 mozilla-central opt test mochitest-8 Android Tegra 250 mozilla-central opt test plain-reftest-1 Android Tegra 250 mozilla-central opt test plain-reftest-2 Android Tegra 250 mozilla-central opt test plain-reftest-3 Android Tegra 250 mozilla-central opt test plain-reftest-4 Android Tegra 250 mozilla-central opt test robocop-1 Android Tegra 250 mozilla-central opt test robocop-2 Android Tegra 250 mozilla-central opt test xpcshell Android Tegra 250 mozilla-central talos remote-tcanvasmark Android Tegra 250 mozilla-central talos remote-tp4m_nochrome Android Tegra 250 mozilla-central talos remote-trobocheck Android Tegra 250 mozilla-central talos remote-trobocheck2 Android Tegra 250 mozilla-central talos remote-trobopan Android Tegra 250 mozilla-central talos remote-troboprovider Android Tegra 250 mozilla-central talos remote-ts Android Tegra 250 mozilla-central talos remote-tsvg Android Tegra 250 mozilla-central talos remote-tsvgx Android X86 mozilla-central build Android X86 mozilla-central nightly Android mozilla-central build Android mozilla-central l10n nightly 1/5 Android mozilla-central l10n nightly 2/5 Android mozilla-central l10n nightly 3/5 Android mozilla-central l10n nightly 4/5 Android mozilla-central l10n nightly 5/5 Android mozilla-central nightly Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-1 Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-2 Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-3 Android no-ionmonkey mozilla-central build When trying to compare staging vs production, I am looked at the results in production here: https://secure.pub.build.mozilla.org/buildapi/revision/mozilla-central/05d3797276d3 The prod issues for revision 05d3797276d3 in mozilla-central were: 1 orange: Android 4.0 Panda mozilla-central opt test crashtest 1 orange: Android Tegra 250 mozilla-central opt test robocop-1 2 red: Android mozilla-central l10n nightly 1/5 2 red: Android mozilla-central l10n nightly 2/5 2 red: Android mozilla-central l10n nightly 3/5 2 red: Android mozilla-central l10n nightly 4/5 2 red: Android mozilla-central l10n nightly 5/5 I am not concerned about l10n nightly, since in my understanding this doesn't run on the device. Therefore I expected to find the same list of builders after running the sendchanges against Staging, and to have the same two oranges as above. This was not the case. Please note the following builders seem to be missing from the prod Builders above: Android 4.0 Panda mozilla-central talos remote-trobocheck Android Tegra 250 mozilla-central talos remote-trobocheck However, their counterparts do exist: Android 4.0 Panda mozilla-central talos remote-trobocheck2 Android Tegra 250 mozilla-central talos remote-trobocheck2 In staging, all 4 of these appear. As you can see, they are not in the list here: https://secure.pub.build.mozilla.org/buildapi/revision/mozilla-central/05d3797276d3 Now on to the first Staging run, on 1 August: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20crashtest/builds/219 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20jsreftest-1/builds/223 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20jsreftest-2/builds/223 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20jsreftest-3/builds/223 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-1/builds/226 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-2/builds/224 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-3/builds/226 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-4/builds/225 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-5/builds/227 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-6/builds/225 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-7/builds/217 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-8/builds/225 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20mochitest-gl/builds/143 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20robocop-1/builds/229 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20robocop-2/builds/221 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20robocop-2/builds/222 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20robocop-2/builds/223 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tp4m_nochrome/builds/29 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-trobocheck/builds/26 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-trobocheck2/builds/25 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-trobopan/builds/30 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-trobopan/builds/31 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-troboprovider/builds/26 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-troboprovider/builds/27 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-ts/builds/27 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/27 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/28 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/29 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/30 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/31 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/32 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvgx/builds/2 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20crashtest/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-1/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-2/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-3/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-1/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-2/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-3/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-4/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-5/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-6/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-7/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-1/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-2/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-3/builds/5 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-4/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-1/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-2/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20xpcshell/builds/4 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20crashtest/builds/3638 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-1/builds/3583 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-2/builds/1794 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-3/builds/3634 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-1/builds/1771 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-2/builds/1759 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-3/builds/3581 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-4/builds/3538 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-5/builds/3612 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-6/builds/3631 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-7/builds/3588 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/3538 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-1/builds/3603 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-2/builds/1788 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-3/builds/3632 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-4/builds/1780 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-1/builds/1843 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-2/builds/3645 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20xpcshell/builds/3595 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tp4m_nochrome/builds/18 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocheck/builds/19 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocheck2/builds/18 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobopan/builds/18 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-troboprovider/builds/20 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-ts/builds/19 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tsvg/builds/19 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tsvgx/builds/2 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20no-ionmonkey%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-1/builds/0 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20no-ionmonkey%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-2/builds/0 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20no-ionmonkey%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-3/builds/0 For some reason, the following Builders appear to be missing: Android 4.0 Panda mozilla-central talos remote-tcanvasmark Android Armv6 mozilla-central build Android Armv6 mozilla-central nightly Android Debug mozilla-central build Android Tegra 250 mozilla-central talos remote-tcanvasmark Android X86 mozilla-central build Android X86 mozilla-central nightly Android mozilla-central build Android mozilla-central l10n nightly 1/5 Android mozilla-central l10n nightly 2/5 Android mozilla-central l10n nightly 3/5 Android mozilla-central l10n nightly 4/5 Android mozilla-central l10n nightly 5/5 Android mozilla-central nightly Android no-ionmonkey mozilla-central build Also, there are several warnings/errors/failures in this list: Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/3538 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-3/builds/3581 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-1/builds/1771 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-2/builds/1759 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-4/builds/3538 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-6/builds/3631 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-7/builds/3588 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/31 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-5/builds/3612 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-5/builds/4 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/30 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20robocop-2/builds/222 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/5 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-4/builds/5 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/29 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-3/builds/4 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-1/builds/4 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/28 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-6/builds/5 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-2/builds/5 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tsvg/builds/27 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20robocop-2/builds/221 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-trobopan/builds/30 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-7/builds/6 Build successful run_script slave lost: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-troboprovider/builds/26 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20crashtest/builds/219 Now on to the next Staging run, on 14 August: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tcanvasmark/builds/517 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20crashtest/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-1/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-2/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-3/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-1/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-2/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-3/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-4/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-5/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-6/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-7/builds/8 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-1/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-2/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-3/builds/7 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-4/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-1/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-2/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20xpcshell/builds/6 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20crashtest/builds/3657 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-1/builds/3603 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-2/builds/1814 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-3/builds/3656 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-1/builds/1791 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-2/builds/1779 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-3/builds/3602 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-4/builds/3558 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-5/builds/3631 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-6/builds/3653 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-7/builds/3607 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/3559 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-1/builds/3625 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-2/builds/1808 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-3/builds/3653 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20plain-reftest-4/builds/1800 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-1/builds/1864 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20robocop-2/builds/3668 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20xpcshell/builds/3617 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tcanvasmark/builds/51 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tp4m_nochrome/builds/70 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocheck2/builds/71 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobopan/builds/70 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-troboprovider/builds/72 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-ts/builds/71 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tsvg/builds/72 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-tsvgx/builds/54 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20no-ionmonkey%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-1/builds/2 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20no-ionmonkey%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-2/builds/2 http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20no-ionmonkey%20Tegra%20250%20mozilla-central%20opt%20test%20jsreftest-3/builds/2 This time, the following Builders appear to be missing: Android 4.0 Panda mozilla-central opt test crashtest Android 4.0 Panda mozilla-central opt test jsreftest-1 Android 4.0 Panda mozilla-central opt test jsreftest-2 Android 4.0 Panda mozilla-central opt test jsreftest-3 Android 4.0 Panda mozilla-central opt test mochitest-1 Android 4.0 Panda mozilla-central opt test mochitest-2 Android 4.0 Panda mozilla-central opt test mochitest-3 Android 4.0 Panda mozilla-central opt test mochitest-4 Android 4.0 Panda mozilla-central opt test mochitest-5 Android 4.0 Panda mozilla-central opt test mochitest-6 Android 4.0 Panda mozilla-central opt test mochitest-7 Android 4.0 Panda mozilla-central opt test mochitest-8 Android 4.0 Panda mozilla-central opt test mochitest-gl Android 4.0 Panda mozilla-central opt test robocop-1 Android 4.0 Panda mozilla-central opt test robocop-2 Android 4.0 Panda mozilla-central talos remote-tp4m_nochrome Android 4.0 Panda mozilla-central talos remote-trobocheck Android 4.0 Panda mozilla-central talos remote-trobocheck2 Android 4.0 Panda mozilla-central talos remote-trobopan Android 4.0 Panda mozilla-central talos remote-troboprovider Android 4.0 Panda mozilla-central talos remote-ts Android 4.0 Panda mozilla-central talos remote-tsvg Android 4.0 Panda mozilla-central talos remote-tsvgx Android Armv6 mozilla-central build Android Armv6 mozilla-central nightly Android Debug mozilla-central build Android Tegra 250 mozilla-central talos remote-trobocheck Android X86 mozilla-central build Android X86 mozilla-central nightly Android mozilla-central build Android mozilla-central l10n nightly 1/5 Android mozilla-central l10n nightly 2/5 Android mozilla-central l10n nightly 3/5 Android mozilla-central l10n nightly 4/5 Android mozilla-central l10n nightly 5/5 Android mozilla-central nightly Android no-ionmonkey mozilla-central build I have no idea at the moment why these do not appear in the staging history, if they existed and disappeared, or if they never got run. In addition, the following had problems/errors/issues: Exception interrupted: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20talos%20remote-tcanvasmark/builds/517 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Armv6%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/7 Warnings mochitest-plain: http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Tegra%20250%20mozilla-central%20opt%20test%20mochitest-8/builds/3559 So to summarize, the Builders that ran in Production, Staging on 1 August and Staging on 14 August all seem to be different, even though they were run from the same sendchanges, or so I believe. Aside from the discrepancies between which Builders actually ran, even for the same Builders that ran all three times, they had different results, despite running against the same revision 05d3797276d3. In this comment (apologies for the length) I have included links to all slave pages which contain e.g. all logs, so that a deeper investigation can be performed - hopefully someone can help me with this that has a better understanding of what the errors/failures/exceptions mean. Thanks, Pete

To fill in one trivial piece of the puzzle, I can confirm that remote-trobocheck was retired in bug 888464 -- you can safely ignore anything related to remote-trobocheck (remote-trobocheck2 continues to run).

Thanks Geoff. I've now run 5 sets of test runs, and can say that I have not been able to find any problems in Staging that seem related to the Watcher upgrade. The differences between builders that got called is explainable based on several factors: the difference in the buildbot repositories between the times the builds ran in production and test (and associated reconfigs that took place), and the settings of the staging build master, with regard to which builders are enabled/disabled. Regarding the test runs themselves, I compared three independent builds: 05d3797276d3, a71cedddadd1, d136c8999d96. All looks to be fine. All staging tegras and pandas were upgraded, and tests were run against both types of devices. I therefore we are now able to proceed to the next step, which I would consider to be to start upgrading some production devices (presumably not all). However, we should align with relevant parties (e.g. Callek, the sherrifs, jmaher, etc) to make sure we are aligned with regard to strategy. Assuming in the end the prod devices get upgraded without issues, and the sherrifs are also happy, we can look at updating the imaging process, so that future re-images will also pick up the new Watcher. Any questions, let me know. Thanks, Pete

fantastic. I would recommend we update pandas, either the talos or unittest ones first. If this needs to be all or none that is fine. I recommend pandas as they can be reimaged easier :) I am not sure if the unittest ones (mozpool managed) or talos (sut_tools/foopy managed) would be easier. Possibly unittests this week, talos next week, and then tegras the week after?

Discussed this with Callek today, we believe we can already start testing the imaging process in Staging, even before we have updated the devices in prod. We will keep the old imaging process working with the old watcher, in case we need to roll back the watcher upgrade in production. Callek is going to have a look at this over the next days, and then we'll meet again to discuss the next steps.

I'm pretty sure this bug can be closed - but I will double check tomorrow if all the parts have been completed.