There is a new version of Watcher (1.16) which should be applied to all production tegras. This has already been applied successfully to all staging devices (see https://bugzilla.mozilla.org/show_bug.cgi?id=876715). The plan is to deploy this to all production tegras and pandas on Friday 13 (!!) September, in the following period (stated in multiple timezones, for convenience): 00:30 - 04:30 PDT 03:30 - 07:30 EDT 09:30 - 13:30 CEST 19:30 - 23:30 NZST If any major issues are discovered, pandas can be reimaged via Mozpool, and tegras can be reimaged using the imaging notebook. In addition, we will be providing a new image for tegras which contains the updated Watcher (1.16) and the new SUT Agent (1.19). Assuming no major issues are found after go-live, these new images can replace the existing images we have, so that future tegra-reimaging will use the new versions of the tools.

Some requests: Sheriffs: please can you confirm that you are ok with this deployment window? All: My current strategy is to have a complete outage during this time for mobile testing - however, if required, I can deploy in 2 batches of masters, so that half the tegras and pandas are available while the other half are being updated, and vice versa. Please let me know if this is required. This would then not require a complete outage, but we would have a longer period of low capacity. Callek/kmoir: can you review my description and check you agree with everything I have said. Callek: can you please confirm how the rollback procedure should be triggered for tegras, and let me know if I need to make special arrangements that the notebook is available, powered on, and connected to the network, at this time, in case nobody is physically present at the time the rollback needs to take place? Callek, could you also provide the bug # of the ticket you raised for getting a new panda image containing the new watcher and sut agent? Thanks all!

I'll defer the sheriff part of this to Ed/Tomcat since the window falls during the bulk of their work day.

I'm even further from Ed/Tomcat's timezone than Ryan.

lgtm As an aside, the mobile test jobs will just queue and then be served when the buildbot masters are up again. Just to clarify that the the nuance behind the word outage :-)

(In reply to Pete Moore [:pete][:pmoore] from comment #1) > Callek/kmoir: can you review my description and check you agree with > everything I have said. I don't see it articulated, but part of the process should be to have someone (relops, or you/us if they tell us how) update the image in mozpool that is "android panda" so that requests from buildbot get the new image, for any that get reimaged after a self-test, etc. (so we don't accidentally downgrade). This should be easy once we test the image update (which is still pending) > Callek: can you please confirm how the rollback procedure should be > triggered for tegras, and let me know if I need to make special arrangements > that the notebook is available, powered on, and connected to the network, at > this time, in case nobody is physically present at the time the rollback > needs to take place? For tegras, I would say the rollback, would be to: (a) Manually Deploy old watcher apk to all reachable devices. (risky, but better than 0 capacity) (b) ask DCOps to reimage ALL tegras with "known good" image in 2 or 3 batches, where we have reduced capacity for each batch. I don't expect rollback to be easy here, but I also don't forsee a reasonable way to mitigate that risk/pain. For pandas rollback would be to: (a) graceful panda masters (b) swap default "android panda" image back to old image (c) manually reimage all android pandas (d) re-enable panda masters > Callek, could you also provide the bug # of the ticket you raised for > getting a new panda image containing the new watcher and sut agent? Bug 914472

If you mean "we're going to close all trees that do mobile, except maybe Try," then that sounds fine to me. Comment 4 could be interpreted as meaning "since jobs will just queue up, no need to close trees," which would only be sort of vaguely true if we didn't coalesce jobs. It may also result in an interesting situation on Try, since we typically carry an 8-10 hour backlog of Tegra jobs into that time of day, and catch up then, so if we don't catch up then... interesting.

Yeah this will need a tree closure, unless we do them in 2-3 batches at a quiet time, IMO.

All, this requires that Relops be involved and a CAB review and a communication out to everyon@mozilla.com if this has the risk of taking a tree down. Please have those planned out for the next week CAB changes before this is implemented.

(In reply to SylvieV from comment #8) Just to make sure the right group gets invited to the table, this requires dcops for the worst case recovery scenario, not relops.

per meeting w/pmoore, coop, callek, joduinn: 1) We're cancelling tomorrow's planned upgrade work, because this time would be during the Oslo-based b2g workweek. We expect early next week to be much lower load, as everyone travels back from the b2g workweek. 2) The nature of this rollout can be done in chunks, which avoids the need for any tree closure / downtime. If this could only be done as one-big-bang-treeclosure, it should have gone through CAB review, like all other tree closures. However, doing in chunks like this will take physically longer, but is worth doing a) avoids tree closures and b) is safer in case of any need to rollback. 3) Here is the revised plan: * On each of Monday / Tues / Wed / Thurs, during 09:30-13:30 CET, pmoore will be taking down one of the 4 tegra-masters per day, and upgrading the foopies+tegras attached to that master. * Based on earlier discussions with sheriffs, we believe doing this upgrade in 4 chunks, during early-morning low load times means we do *not* need a tree closure for this. Edmorley/ryanvm, please confirm? * Doing the upgrade on Pandas will be different, because of MozPool, and does not need to be at same time as the above tegra work. More details once we see how the tegra rollout goes.

Given the times being discussed, I'll defer back to comment 2 :)

wfm

ryamvm, ed: glad that proposal works for you both, and avoids a tree closure! dmoore: I know there were various in-person meetings about this yesterday, and I believe this is works for you... but I've I just noticed you're not cc'd, so adding you.

Latest plan can be seen here: http://petemoore.tumblr.com/post/61010673496/runbook-for-tegra-watcher-rollout-16-19-sep-2013 Here is a copy for convenience: Schedule Mon 16 Sep 2013 http://buildbot-master10.build.mtv1.mozilla.com:8201/ * foopy28 - all devices * foopy29 - all devices *except* tegra-342 * foopy30 - all devices *except* tegra-349 * foopy31 - all devices * foopy32 - all devices Tue 17 Sep 2013 http://buildbot-master19.build.mtv1.mozilla.com:8201/ Wed 18 Sep 2013 http://buildbot-master20.build.mtv1.mozilla.com:8201/ Thu 19 Sep 2013 http://buildbot-master22.build.mtv1.mozilla.com:8201/ Steps * Inform sheriffs via #releng that buildbot master will be shutdown * Send email to confirm that buildbot master will be shutdown (release@mozilla.com) * Update bug 914302 to say deployment is starting for this master * Click on “Clean Shutdown” * Wait until graceful shutdown is completed * Upgrade all tegras * Start master using fabric, e.g. inside virtual env containing fabric: python <full path to>/manage_masters.py -f <full path to>/production-masters.json -H bm10-tests1-tegra start * Monitor results * Advise sheriffs that buildbot master is back again, and ask to be notified in case of problems * Email release@mozilla.com to say all is ok again * Update bug 914302 to say master is completed

Dear all, * Starting phase 1 now (tegras on buildbot-master10) * No outage - but buildbot-master10 will be down while upgrading Thanks, Pete

buildbot-master10 down, update of tegras starting

All tegras are now currently being updated in parallel on foopy28, foopy29, foopy30, foopy31, foopy32 with the following exceptions: * tegra-342 on foopy29 is configured to connect to buildbot-master20.build.mtv1.mozilla.com (not sure why - will investigate) * tegra-349 on foopy30 is disabled in slavealloc I will post an update with the results when completed.

Tegras upgraded, now collecting results to check for any failures, etc.

These upgrades were all successful: SUCCESS: tegra-286 (foopy30) SUCCESS: tegra-308 (foopy30) SUCCESS: tegra-309 (foopy28) SUCCESS: tegra-313 (foopy28) SUCCESS: tegra-315 (foopy30) SUCCESS: tegra-316 (foopy28) SUCCESS: tegra-317 (foopy28) SUCCESS: tegra-318 (foopy28) SUCCESS: tegra-321 (foopy28) SUCCESS: tegra-322 (foopy31) SUCCESS: tegra-324 (foopy29) SUCCESS: tegra-326 (foopy29) SUCCESS: tegra-327 (foopy31) SUCCESS: tegra-329 (foopy31) SUCCESS: tegra-330 (foopy29) SUCCESS: tegra-331 (foopy29) SUCCESS: tegra-332 (foopy29) SUCCESS: tegra-334 (foopy29) SUCCESS: tegra-335 (foopy31) SUCCESS: tegra-338 (foopy29) SUCCESS: tegra-340 (foopy29) SUCCESS: tegra-343 (foopy29) SUCCESS: tegra-344 (foopy31) SUCCESS: tegra-346 (foopy29) SUCCESS: tegra-347 (foopy30) SUCCESS: tegra-348 (foopy31) SUCCESS: tegra-351 (foopy30) SUCCESS: tegra-352 (foopy32) SUCCESS: tegra-353 (foopy32) SUCCESS: tegra-354 (foopy32) SUCCESS: tegra-355 (foopy32) SUCCESS: tegra-357 (foopy32) SUCCESS: tegra-358 (foopy32) SUCCESS: tegra-359 (foopy30) SUCCESS: tegra-362 (foopy30) SUCCESS: tegra-363 (foopy32) SUCCESS: tegra-364 (foopy30) SUCCESS: tegra-367 (foopy30) SUCCESS: tegra-369 (foopy32) The following tegra upgrades need to be investigated, since they did not log a successful upgrade: INVESTIGATE: tegra-307 (foopy28) INVESTIGATE: tegra-310 (foopy28) INVESTIGATE: tegra-311 (foopy28) INVESTIGATE: tegra-312 (foopy28) INVESTIGATE: tegra-314 (foopy28) INVESTIGATE: tegra-319 (foopy28) INVESTIGATE: tegra-320 (foopy31) INVESTIGATE: tegra-323 (foopy28) INVESTIGATE: tegra-325 (foopy29) INVESTIGATE: tegra-328 (foopy31) INVESTIGATE: tegra-333 (foopy31) INVESTIGATE: tegra-336 (foopy31) INVESTIGATE: tegra-337 (foopy31) INVESTIGATE: tegra-339 (foopy29) INVESTIGATE: tegra-341 (foopy31) INVESTIGATE: tegra-345 (foopy31) INVESTIGATE: tegra-350 (foopy32) INVESTIGATE: tegra-356 (foopy30) INVESTIGATE: tegra-360 (foopy32) INVESTIGATE: tegra-361 (foopy30) INVESTIGATE: tegra-365 (foopy30)

Foopy 28 first investigation results ==================================== INVESTIGATE: tegra-307 (foopy28) Could not connect Buildbot slave not running No error.flg (strange) INVESTIGATE: tegra-310 (foopy28) Upgraded, but timed out waiting for device to come back up Buildbot slave not running error.flg says "Remote Device Error: unable to write to sdcard" INVESTIGATE: tegra-311 (foopy28) Upgraded, but timed out waiting for device to come back up Buildbot slave now running again (device passed verify.py) INVESTIGATE: tegra-312 (foopy28) Could not connect Buildbot slave now running again (device passed verify.py) INVESTIGATE: tegra-314 (foopy28) Upgraded, but timed out waiting for device to come back up Buildbot slave not running error.flg says "Remote Device Error: unable to write to sdcard" INVESTIGATE: tegra-319 (foopy28) Upgraded, but timed out waiting for device to come back up Buildbot slave now running again (device passed verify.py) INVESTIGATE: tegra-323 (foopy28) Upgraded, but timed out waiting for device to come back up Buildbot slave now running again (device passed verify.py)

Foopy 28 further results ======================== INVESTIGATE: tegra-307 (foopy28) Could not connect Buildbot slave not running error.flg: Unable to ping device after 5 attempts INVESTIGATE: tegra-310 (foopy28) Upgrade ok Buildbot not running error.flg: Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-311 (foopy28) Intermittent connection problems, but device upgraded ok Buildbot currently running INVESTIGATE: tegra-312 (foopy28) **Not upgraded** Buildbot running INVESTIGATE: tegra-314 (foopy28) Upgrade ok Buildbot not running error.flg: Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-319 (foopy28) Now ok, upgrade worked, buildbot running INVESTIGATE: tegra-323 (foopy28) Now ok, upgrade worked, buildbot running

Complete results: Foopy 28 investigation results ============================== INVESTIGATE: tegra-307 (foopy28) Could not connect error.flg: Unable to ping device after 5 attempts INVESTIGATE: tegra-310 (foopy28) Upgrade ok error.flg: Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-311 (foopy28) Intermittent connection problems, but device upgraded ok Buildbot currently running INVESTIGATE: tegra-312 (foopy28) **Not upgraded** Buildbot running INVESTIGATE: tegra-314 (foopy28) Upgrade ok error.flg: Remote Device Error: unable to write to sdcard Foopy 29 investigation results ============================== INVESTIGATE: tegra-324 (foopy29) Watcher upgraded ok Remote Device Error: unable to write to sdcard Buildbot not running, no error.flg yet INVESTIGATE: tegra-325 (foopy29) Automation Error: Unable to connect to device after 5 attempts INVESTIGATE: tegra-339 (foopy29) Watcher upgraded ok Remote Device Error: unable to write to sdcard Foopy 30 investigation results ============================== INVESTIGATE: tegra-356 (foopy30) Automation Error: Unable to connect to device after 5 attempts INVESTIGATE: tegra-361 (foopy30) Watcher upgraded ok Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-365 (foopy30) Watcher upgraded ok Remote Device Error: unable to write to sdcard Foopy 31 investigation results ============================== INVESTIGATE: tegra-320 (foopy31) Not upgraded INVESTIGATE: tegra-328 (foopy31) Cannot connect No error.flg at the moment INVESTIGATE: tegra-333 (foopy31) - all ok again INVESTIGATE: tegra-336 (foopy31) - all ok again INVESTIGATE: tegra-337 (foopy31) Watcher upgrade ok Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-341 (foopy31) Watcher upgrade ok Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-345 (foopy31) Cannot connect No error.flg at the moment Foopy 32 investigation results ============================== INVESTIGATE: tegra-350 (foopy32) Watcher upgrade ok Remote Device Error: unable to write to sdcard ***NEW***: tegra-358 (foopy32) Watcher upgrade ok Remote Device Error: unable to write to sdcard INVESTIGATE: tegra-360 (foopy32) Cannot connect No error.flg at the moment

I don't consider any of the above issues to be blocking, so will restart buildbot master 10.

http://buildbot-master10.build.mtv1.mozilla.com:8201/ up and running again, notified #releng. Work completed for day 1/4, with the exception of the individual tegra issues described in connect 22 above, which will be handed as BAU. Tomorrow - Tue Sep 17 (Day 2/4) will be the upgrade of tegras attached to the next tegra buildbot master (http://buildbot-master19.build.mtv1.mozilla.com:8201/) - starting at 9:30 CEST...

REGARDING ***PANDA*** ROLLOUT PLANS =================================== Regarding panda roll out - I have discussed this with kmoir and jmaher, and bug 829211 should be landing in the next week or so, so I would propose we roll out to pandas after bug 829211 lands. The biggest motivator for this, is that once all pandas are mozpool-managed, we simply have to update the image name in mozharness to the new mozpool panda image that contains the new watcher, and this should take care that all the pandas then run with the correct image. It is also trivial to rollback the change, since it would be a case of reverting the change.

Dear all, * Starting phase 2 now (tegras on buildbot-master19) * No outage - but buildbot-master19 will be down while upgrading Thanks, Pete

All tegras on buildbot-master19 patched, checking results...

Everything it taking a little longer today - hope to be all done in the next hour.

Upgrade problems on foopy27 =========================== tegra-250: Cannot connect to device Upgrade problems on foopy111 ============================ No upgrade problems found Upgrade problems on foopy115 ============================ No upgrade problems found Upgrade problems on foopy117 ============================ tegra-108: Cannot connect to device tegra-113: Cannot connect to device tegra-115: Cannot connect to device Upgrade problems on foopy121 ============================ No upgrade problems found Upgrade problems on foopy123... ============================ tegra-223: Cannot connect to device tegra-229: Cannot connect to device Other notes =========== tegra-228 (foopy123) ========= Not upgraded - intentional - not a problem tegra-261 (foopy123) ========= Not upgraded - intentional - not a problem

buildbot-master19 restarted.... Watching http://buildbot-master19.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1

After starting up buildbot master and puppet on foopies, and watch_devices.sh on foopies, I'm now additionally having issues connecting to the following three tegras: tegra-087 on foopy115 tegra-177 and tegra-189 on foopy121 I could connect to these devices before the upgrade. I'm not sure at the moment what the problem is.

Correction to comment above in comment 31: I mean, I could connect to them after the upgrade, before starting up buildbot master and slaves...

Strangely, we seem to have made a nett win today. See results below: Tegras that had no error.flg before or after the watcher upgrade on buildbot-master19 today: tegra-031 tegra-032 tegra-035 tegra-036 tegra-037 tegra-038 tegra-039 tegra-078 tegra-080 tegra-081 tegra-082 tegra-083 tegra-085 tegra-086 tegra-088 tegra-089 tegra-105 tegra-107 tegra-109 tegra-110 tegra-111 tegra-112 tegra-117 tegra-146 tegra-178 tegra-179 tegra-181 tegra-182 tegra-183 tegra-188 tegra-190 tegra-194 tegra-199 tegra-200 tegra-225 tegra-227 tegra-231 tegra-233 tegra-248 tegra-249 tegra-251 tegra-252 tegra-253 tegra-254 tegra-255 tegra-256 tegra-257 tegra-258 tegra-259 tegra-260 Tegras that previously had an error.flg file, before the upgrade, and still have one, afterwards: tegra-064 tegra-087 tegra-108 tegra-113 tegra-114 tegra-177 tegra-189 tegra-220 tegra-221 tegra-223 tegra-229 tegra-232 Tegras that previously had error.flg files, that were cleared after the Watcher upgrade (!!!!) tegra-079 tegra-084 tegra-106 tegra-116 tegra-173 tegra-174 tegra-187 tegra-196 Tegras that previously did not have an error.flg file, that now, after the upgrade, have one: tegra-115 tegra-250

There are 6 tegras that don't have an error.flg file, but that are in any case not connected when I look at http://buildbot-master19.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1 - so I will investigate these now: tegra-032 tegra-111 tegra-116 tegra-117 tegra-233 tegra-248

So of these 6 non-connected tegras, it turns out 4 were rebooting, and are connected again. The other two now have error.flg files. See below: tegra-032: was rebooting, now is connected again. tegra-111: was rebooting, now is connected again. tegra-116: Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/writetest 11865'; err='Push failed!' error.flg: Remote Device Error: unable to write to sdcard tegra-117: 09/17/2013 08:15:08: INFO: INFO: Got expected SUTAgent version '1.19' 09/17/2013 08:15:08: INFO: Uninstalling org.mozilla.fennec... 09/17/2013 08:15:08: INFO: Waiting for device to come back... 09/17/2013 08:17:08: INFO: Try 1 09/17/2013 08:18:04: INFO: devroot None 2013-09-17 08:20:02 -- *** ERROR *** failed to aquire lockfile error.flg: Remote Device Error: Unhandled exception in cleanupDevice tegra-233: was rebooting, now is connected again. tegra-248: was rebooting, now is connected again.

I'm calling a day today, will continue tomorrow. :)

Dear all, * Starting phase 3 of https://bugzilla.mozilla.org/show_bug.cgi?id=914302 * Updating all buildbot-master20 tegras to new Watcher 1.16 (production) * No outage - but buildbot-master20 will be down during this upgrade. Thanks, Pete

Summary report for day 3/4 (buildbot-master20) ========================== * only one device failed to upgrade: tegra-076 mozdevice.devicemanager.DMError: Did not successfully run command ['dd', 'if=/mnt/sdcard/tests/Watcher.1.16.apk', 'of=/data/local/watcher.apk'] (output: '', retval: 'None') * two devices upgraded, but did not reboot successfully (so are currently unreachable): tegra-219 tegra-300 * nine devices could not be upgraded, because they could not be reached: tegra-050 tegra-066 tegra-073 tegra-074 tegra-090 tegra-150 tegra-166 tegra-171 tegra-279 Will now restart puppet, watch_devices.sh, and buildbot master 20...

buildbot-master20 is back up and running, I'm monitoring the tegras here: http://buildbot-master20.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1 I will file bugs for any tegras that need to be reimaged. Thanks, Pete

Dear all, * Phase 4 is underway (tegras on buildbot-master22) * No tree outage - but buildbot-master22 is down while I am upgrading the tegras Thanks, Pete

Summary report for day 4/4 (buildbot-master22) ========================== * two devices failed to upgrade: tegra-264@foopy126 tegra-270@foopy126 mozdevice.devicemanager.DMError: Did not successfully run command ['dd', 'if=/mnt/sdcard/tests/Watcher.1.16.apk', 'of=/data/local/watcher.apk'] (output: '', retval: 'None') * four devices could not be upgraded, because they could not be reached: tegra-130@foopy119 tegra-218@foopy122 tegra-242@foopy124 tegra-269@foopy126

Started up buildbot-master22 again, and am monitoring http://buildbot-master22.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1 This bug is now complete. I will follow up with a separate bug for tegras that still require action, such as a new image.

Reopening, because this ticket was also to encompass panda upgrades (whoops) .... Thanks jmaher for reminding me!

Just generating a final list of tegras that had issues, and will be adding them to bug 918162...

Tegras that we cannot connect to, to see if the Watcher upgraded ok =================================================================== foopy109: tegra-184: Cannot connect to device foopy109: tegra-185: Cannot connect to device foopy109: tegra-224: Cannot connect to device foopy110: tegra-020: Cannot connect to device foopy112: tegra-050: Cannot connect to device foopy112: tegra-090: Cannot connect to device foopy114: tegra-066: Cannot connect to device foopy114: tegra-073: Cannot connect to device foopy114: tegra-074: Cannot connect to device foopy117: tegra-108: Cannot connect to device foopy117: tegra-113: Cannot connect to device foopy118: tegra-150: Cannot connect to device foopy118: tegra-152: Cannot connect to device foopy119: tegra-130: Cannot connect to device foopy120: tegra-166: Cannot connect to device foopy120: tegra-171: Cannot connect to device foopy122: tegra-218: Cannot connect to device foopy123: tegra-223: Cannot connect to device foopy123: tegra-228: Cannot connect to device foopy123: tegra-229: Cannot connect to device foopy124: tegra-239: Cannot connect to device foopy125: tegra-293: Cannot connect to device foopy126: tegra-269: Cannot connect to device foopy126: tegra-270: Cannot connect to device foopy128: tegra-279: Cannot connect to device foopy28: tegra-307: Cannot connect to device foopy28: tegra-309: Cannot connect to device foopy29: tegra-338: Cannot connect to device foopy29: tegra-339: Cannot connect to device foopy30: tegra-349: Cannot connect to device foopy30: tegra-356: Cannot connect to device foopy30: tegra-364: Cannot connect to device foopy31: tegra-328: Cannot connect to device foopy31: tegra-341: Cannot connect to device foopy31: tegra-345: Cannot connect to device foopy32: tegra-360: Cannot connect to device foopy34: panda-0046: Cannot connect to device foopy34: panda-0047: Cannot connect to device foopy34: panda-0048: Cannot connect to device foopy34: panda-0049: Cannot connect to device foopy34: panda-0050: Cannot connect to device foopy34: panda-0051: Cannot connect to device foopy34: panda-0052: Cannot connect to device foopy34: panda-0053: Cannot connect to device foopy34: panda-0054: Cannot connect to device foopy34: panda-0055: Cannot connect to device foopy34: panda-0056: Cannot connect to device foopy34: panda-0057: Cannot connect to device foopy35: panda-0058: Cannot connect to device foopy35: panda-0059: Cannot connect to device foopy35: panda-0060: Cannot connect to device foopy35: panda-0061: Cannot connect to device foopy35: panda-0062: Cannot connect to device foopy35: panda-0063: Cannot connect to device foopy35: panda-0064: Cannot connect to device foopy35: panda-0065: Cannot connect to device foopy35: panda-0066: Cannot connect to device foopy35: panda-0067: Cannot connect to device foopy35: panda-0068: Cannot connect to device foopy35: panda-0069: Cannot connect to device Devices that do not have the upgraded Watcher ============================================= foopy109: tegra-010: Not upgraded foopy109: tegra-011: Not upgraded foopy109: tegra-012: Not upgraded foopy109: tegra-013: Not upgraded foopy110: tegra-014: Not upgraded foopy110: tegra-015: Not upgraded foopy110: tegra-016: Not upgraded foopy110: tegra-017: Not upgraded foopy110: tegra-018: Not upgraded foopy110: tegra-019: Not upgraded foopy110: tegra-021: Not upgraded foopy110: tegra-022: Not upgraded foopy110: tegra-023: Not upgraded foopy110: tegra-025: Not upgraded foopy110: tegra-026: Not upgraded foopy110: tegra-027: Not upgraded foopy110: tegra-028: Not upgraded foopy110: tegra-029: Not upgraded foopy112: tegra-042: Not upgraded foopy112: tegra-051: Not upgraded foopy113: tegra-053: Not upgraded foopy113: tegra-060: Not upgraded foopy114: tegra-076: Not upgraded foopy118: tegra-155: Not upgraded foopy119: tegra-142: Not upgraded foopy124: tegra-242: Not upgraded foopy124: tegra-246: Not upgraded foopy126: tegra-264: Not upgraded foopy28: tegra-312: Not upgraded foopy29: tegra-325: Not upgraded foopy31: tegra-320: Not upgraded Devices that were successfully upgraded ======================================= foopy109: tegra-288: Watcher upgraded foopy111: tegra-031: Watcher upgraded foopy111: tegra-032: Watcher upgraded foopy111: tegra-035: Watcher upgraded foopy111: tegra-036: Watcher upgraded foopy111: tegra-037: Watcher upgraded foopy111: tegra-038: Watcher upgraded foopy111: tegra-039: Watcher upgraded foopy111: tegra-064: Watcher upgraded foopy111: tegra-078: Watcher upgraded foopy111: tegra-199: Watcher upgraded foopy111: tegra-200: Watcher upgraded foopy112: tegra-040: Watcher upgraded foopy112: tegra-041: Watcher upgraded foopy112: tegra-045: Watcher upgraded foopy112: tegra-046: Watcher upgraded foopy112: tegra-047: Watcher upgraded foopy112: tegra-048: Watcher upgraded foopy112: tegra-052: Watcher upgraded foopy113: tegra-054: Watcher upgraded foopy113: tegra-055: Watcher upgraded foopy113: tegra-056: Watcher upgraded foopy113: tegra-057: Watcher upgraded foopy113: tegra-058: Watcher upgraded foopy113: tegra-059: Watcher upgraded foopy113: tegra-061: Watcher upgraded foopy113: tegra-062: Watcher upgraded foopy113: tegra-063: Watcher upgraded foopy114: tegra-067: Watcher upgraded foopy114: tegra-068: Watcher upgraded foopy114: tegra-070: Watcher upgraded foopy114: tegra-071: Watcher upgraded foopy114: tegra-072: Watcher upgraded foopy114: tegra-091: Watcher upgraded foopy114: tegra-219: Watcher upgraded foopy115: tegra-079: Watcher upgraded foopy115: tegra-080: Watcher upgraded foopy115: tegra-081: Watcher upgraded foopy115: tegra-082: Watcher upgraded foopy115: tegra-083: Watcher upgraded foopy115: tegra-084: Watcher upgraded foopy115: tegra-085: Watcher upgraded foopy115: tegra-086: Watcher upgraded foopy115: tegra-087: Watcher upgraded foopy115: tegra-088: Watcher upgraded foopy115: tegra-089: Watcher upgraded foopy116: tegra-092: Watcher upgraded foopy116: tegra-093: Watcher upgraded foopy116: tegra-094: Watcher upgraded foopy116: tegra-095: Watcher upgraded foopy116: tegra-096: Watcher upgraded foopy116: tegra-097: Watcher upgraded foopy116: tegra-098: Watcher upgraded foopy116: tegra-099: Watcher upgraded foopy116: tegra-100: Watcher upgraded foopy116: tegra-101: Watcher upgraded foopy116: tegra-102: Watcher upgraded foopy116: tegra-103: Watcher upgraded foopy116: tegra-104: Watcher upgraded foopy117: tegra-105: Watcher upgraded foopy117: tegra-106: Watcher upgraded foopy117: tegra-107: Watcher upgraded foopy117: tegra-109: Watcher upgraded foopy117: tegra-110: Watcher upgraded foopy117: tegra-111: Watcher upgraded foopy117: tegra-112: Watcher upgraded foopy117: tegra-114: Watcher upgraded foopy117: tegra-115: Watcher upgraded foopy117: tegra-116: Watcher upgraded foopy117: tegra-117: Watcher upgraded foopy118: tegra-118: Watcher upgraded foopy118: tegra-119: Watcher upgraded foopy118: tegra-120: Watcher upgraded foopy118: tegra-121: Watcher upgraded foopy118: tegra-145: Watcher upgraded foopy118: tegra-148: Watcher upgraded foopy118: tegra-149: Watcher upgraded foopy118: tegra-151: Watcher upgraded foopy118: tegra-154: Watcher upgraded foopy118: tegra-157: Watcher upgraded foopy119: tegra-129: Watcher upgraded foopy119: tegra-132: Watcher upgraded foopy119: tegra-133: Watcher upgraded foopy119: tegra-134: Watcher upgraded foopy119: tegra-135: Watcher upgraded foopy119: tegra-136: Watcher upgraded foopy119: tegra-138: Watcher upgraded foopy119: tegra-139: Watcher upgraded foopy119: tegra-140: Watcher upgraded foopy119: tegra-141: Watcher upgraded foopy119: tegra-144: Watcher upgraded foopy120: tegra-158: Watcher upgraded foopy120: tegra-159: Watcher upgraded foopy120: tegra-160: Watcher upgraded foopy120: tegra-162: Watcher upgraded foopy120: tegra-163: Watcher upgraded foopy120: tegra-164: Watcher upgraded foopy120: tegra-165: Watcher upgraded foopy120: tegra-167: Watcher upgraded foopy120: tegra-168: Watcher upgraded foopy120: tegra-169: Watcher upgraded foopy120: tegra-170: Watcher upgraded foopy121: tegra-173: Watcher upgraded foopy121: tegra-174: Watcher upgraded foopy121: tegra-177: Watcher upgraded foopy121: tegra-178: Watcher upgraded foopy121: tegra-179: Watcher upgraded foopy121: tegra-181: Watcher upgraded foopy121: tegra-182: Watcher upgraded foopy121: tegra-183: Watcher upgraded foopy121: tegra-187: Watcher upgraded foopy121: tegra-188: Watcher upgraded foopy121: tegra-189: Watcher upgraded foopy121: tegra-190: Watcher upgraded foopy121: tegra-196: Watcher upgraded foopy122: tegra-206: Watcher upgraded foopy122: tegra-207: Watcher upgraded foopy122: tegra-208: Watcher upgraded foopy122: tegra-209: Watcher upgraded foopy122: tegra-210: Watcher upgraded foopy122: tegra-211: Watcher upgraded foopy122: tegra-212: Watcher upgraded foopy122: tegra-213: Watcher upgraded foopy122: tegra-214: Watcher upgraded foopy122: tegra-215: Watcher upgraded foopy122: tegra-216: Watcher upgraded foopy122: tegra-217: Watcher upgraded foopy123: tegra-146: Watcher upgraded foopy123: tegra-194: Watcher upgraded foopy123: tegra-220: Watcher upgraded foopy123: tegra-221: Watcher upgraded foopy123: tegra-225: Watcher upgraded foopy123: tegra-227: Watcher upgraded foopy123: tegra-231: Watcher upgraded foopy123: tegra-232: Watcher upgraded foopy123: tegra-233: Watcher upgraded foopy123: tegra-261: Watcher upgraded foopy124: tegra-234: Watcher upgraded foopy124: tegra-235: Watcher upgraded foopy124: tegra-236: Watcher upgraded foopy124: tegra-237: Watcher upgraded foopy124: tegra-238: Watcher upgraded foopy124: tegra-240: Watcher upgraded foopy124: tegra-243: Watcher upgraded foopy124: tegra-244: Watcher upgraded foopy124: tegra-245: Watcher upgraded foopy124: tegra-247: Watcher upgraded foopy125: tegra-290: Watcher upgraded foopy125: tegra-294: Watcher upgraded foopy125: tegra-295: Watcher upgraded foopy125: tegra-296: Watcher upgraded foopy125: tegra-297: Watcher upgraded foopy125: tegra-298: Watcher upgraded foopy125: tegra-299: Watcher upgraded foopy125: tegra-300: Watcher upgraded foopy125: tegra-302: Watcher upgraded foopy125: tegra-304: Watcher upgraded foopy125: tegra-305: Watcher upgraded foopy125: tegra-306: Watcher upgraded foopy126: tegra-195: Watcher upgraded foopy126: tegra-262: Watcher upgraded foopy126: tegra-263: Watcher upgraded foopy126: tegra-265: Watcher upgraded foopy126: tegra-266: Watcher upgraded foopy126: tegra-267: Watcher upgraded foopy126: tegra-271: Watcher upgraded foopy126: tegra-272: Watcher upgraded foopy126: tegra-273: Watcher upgraded foopy126: tegra-274: Watcher upgraded foopy128: tegra-191: Watcher upgraded foopy128: tegra-192: Watcher upgraded foopy128: tegra-201: Watcher upgraded foopy128: tegra-276: Watcher upgraded foopy128: tegra-277: Watcher upgraded foopy128: tegra-278: Watcher upgraded foopy128: tegra-280: Watcher upgraded foopy128: tegra-281: Watcher upgraded foopy128: tegra-282: Watcher upgraded foopy128: tegra-283: Watcher upgraded foopy128: tegra-284: Watcher upgraded foopy128: tegra-285: Watcher upgraded foopy27: tegra-248: Watcher upgraded foopy27: tegra-249: Watcher upgraded foopy27: tegra-250: Watcher upgraded foopy27: tegra-251: Watcher upgraded foopy27: tegra-252: Watcher upgraded foopy27: tegra-253: Watcher upgraded foopy27: tegra-254: Watcher upgraded foopy27: tegra-255: Watcher upgraded foopy27: tegra-256: Watcher upgraded foopy27: tegra-257: Watcher upgraded foopy27: tegra-258: Watcher upgraded foopy27: tegra-259: Watcher upgraded foopy27: tegra-260: Watcher upgraded foopy28: tegra-310: Watcher upgraded foopy28: tegra-311: Watcher upgraded foopy28: tegra-313: Watcher upgraded foopy28: tegra-314: Watcher upgraded foopy28: tegra-316: Watcher upgraded foopy28: tegra-317: Watcher upgraded foopy28: tegra-318: Watcher upgraded foopy28: tegra-319: Watcher upgraded foopy28: tegra-321: Watcher upgraded foopy28: tegra-323: Watcher upgraded foopy29: tegra-324: Watcher upgraded foopy29: tegra-326: Watcher upgraded foopy29: tegra-330: Watcher upgraded foopy29: tegra-331: Watcher upgraded foopy29: tegra-332: Watcher upgraded foopy29: tegra-334: Watcher upgraded foopy29: tegra-340: Watcher upgraded foopy29: tegra-342: Watcher upgraded foopy29: tegra-343: Watcher upgraded foopy29: tegra-346: Watcher upgraded foopy30: tegra-286: Watcher upgraded foopy30: tegra-308: Watcher upgraded foopy30: tegra-315: Watcher upgraded foopy30: tegra-347: Watcher upgraded foopy30: tegra-351: Watcher upgraded foopy30: tegra-359: Watcher upgraded foopy30: tegra-361: Watcher upgraded foopy30: tegra-362: Watcher upgraded foopy30: tegra-365: Watcher upgraded foopy30: tegra-367: Watcher upgraded foopy31: tegra-322: Watcher upgraded foopy31: tegra-327: Watcher upgraded foopy31: tegra-329: Watcher upgraded foopy31: tegra-333: Watcher upgraded foopy31: tegra-335: Watcher upgraded foopy31: tegra-336: Watcher upgraded foopy31: tegra-337: Watcher upgraded foopy31: tegra-344: Watcher upgraded foopy31: tegra-348: Watcher upgraded foopy32: tegra-350: Watcher upgraded foopy32: tegra-352: Watcher upgraded foopy32: tegra-353: Watcher upgraded foopy32: tegra-354: Watcher upgraded foopy32: tegra-355: Watcher upgraded foopy32: tegra-357: Watcher upgraded foopy32: tegra-358: Watcher upgraded foopy32: tegra-363: Watcher upgraded foopy32: tegra-369: Watcher upgraded

This work is now completed. Closing bug.