Over in Bug 1064004, I'm trying to get additional Android instrumentation test suites running in automation. This is intended to be the analog of Bug 961138, which landed a mozharness suite for web-platform-tests.

If I understand jlund correctly, once I have a new suite running on cedar, I can develop the mozharness script in the default branch (only merging to the production branch when we land). In theory, it is not hard to develop a mozharness script that does the right thing locally; in practice, I'm not sure that's true. I saw no "mozharness try" builds in the web-platform-tests example ticket. jlund, can you verify that what I say above is correct; and can you redirect me to somebody who might know how to iterate on mozharness scripts?

'mozharness try' is 'ash-mozharness'. the ash branch uses ash-mozharness for all its jobs. it never merges to mainline mozharness. cedar uses mozharness 'default'. default is merged nearly every day. if your script is new, you can develop it on mainline mozharness as it won't affect anything bar your test suite running on cedar.

Ah, copy-and-paste, but no sense waiting to get feedback on this stuff. If my understanding is correct, this should try to run instrumentation/runinstrumentation.py --suite {browser,background} I intend for runinstrumentation.py to get its manifest and APK files from the test bundle (rather than uploading another APK). I'll need to add a section "instrumentation_options" to testing/config/mozharness/android_panda_config.py to pass IP addresses, etc through. I have no idea how one can have any confidence in any of this before it's in automation; if there's some /reasonable/ way to test, I am happy to do so.

Comment on attachment 8486958 [details] [diff] [review] 1064012.patch Review of attachment 8486958 [details] [diff] [review]: ----------------------------------------------------------------- This looks right to me. To get some sense of whether this is going to work before checking in, I suggest pushing to http://hg.mozilla.org/build/ash-mozharness/. That will only affect the ash tree. As soon as you push, re-trigger some existing Android 4.0 jobs on ash. Without buildbot-config changes, nothing will trigger your instrumentation changes, but you should still be able to verify that you have done no harm to the existing tests. Let's check in with kmoir too... ::: configs/android/android_panda_releng.py @@ +15,5 @@ > "reftest": "remotereftest.py", > "crashtest": "remotereftest.py", > "jsreftest": "remotereftest.py", > "robocop": "runtestsremote.py", > + "instrumentation": "runinstrumentation.py", Have you written runinstrumentation.py yet? ::: scripts/android_panda.py @@ +405,5 @@ > abs_dirs['abs_work_dir'], 'hostutils') > dirs['abs_robocop_dir'] = os.path.join( > dirs['abs_test_install_dir'], 'mochitest') > + dirs['abs_instrumentation_dir'] = os.path.join( > + dirs['abs_test_install_dir'], 'instrumentation') Be aware that most of the other directories in abs_test_install_dir (like mochitest) are created by unzipping the tests.zip file. I'm not too concerned right now because I don't think you have anything referencing abs_instrumentation_dir, but at some point you should probably add "instrumentation" to the tests.zip.

(In reply to Geoff Brown [:gbrown] (PTO Sept 15 - Oct 7) from comment #4) > Comment on attachment 8486958 [details] [diff] [review] > 1064012.patch > > Review of attachment 8486958 [details] [diff] [review]: > ----------------------------------------------------------------- > > This looks right to me. > > To get some sense of whether this is going to work before checking in, I > suggest pushing to http://hg.mozilla.org/build/ash-mozharness/. That will > only affect the ash tree. As soon as you push, re-trigger some existing > Android 4.0 jobs on ash. Without buildbot-config changes, nothing will > trigger your instrumentation changes, but you should still be able to verify > that you have done no harm to the existing tests. Pushed to ash-mozharness: http://hg.mozilla.org/build/ash-mozharness/rev/5a0010c7a24d Retriggered the following jobs: Retrigger of Android 2.3 JSReftest 1 opt requested. Retrigger of Android 2.3 JSReftest 2 opt requested. Retrigger of Android 2.3 Reftest 1 opt requested. Retrigger of Android 2.3 Reftest 2 opt requested. Retrigger of Android 2.3 Crashtest 1 opt requested. on push: https://tbpl.mozilla.org/?tree=Ash&rev=6b8da5940f74 > Let's check in with kmoir too... > > ::: configs/android/android_panda_releng.py > @@ +15,5 @@ > > "reftest": "remotereftest.py", > > "crashtest": "remotereftest.py", > > "jsreftest": "remotereftest.py", > > "robocop": "runtestsremote.py", > > + "instrumentation": "runinstrumentation.py", > > Have you written runinstrumentation.py yet? Nope! This is the part that's easiest for me, since I can iterate in tree. > ::: scripts/android_panda.py > @@ +405,5 @@ > > abs_dirs['abs_work_dir'], 'hostutils') > > dirs['abs_robocop_dir'] = os.path.join( > > dirs['abs_test_install_dir'], 'mochitest') > > + dirs['abs_instrumentation_dir'] = os.path.join( > > + dirs['abs_test_install_dir'], 'instrumentation') > > Be aware that most of the other directories in abs_test_install_dir (like > mochitest) are created by unzipping the tests.zip file. I'm not too > concerned right now because I don't think you have anything referencing > abs_instrumentation_dir, but at some point you should probably add > "instrumentation" to the tests.zip. This is by design. We shouldn't be baking APK-specific logic into buildbots or even mozharness. In fact, I'd like to bake robocop.apk into tests.zip; but that's another discussion.

> Retriggered the following jobs: > > Retrigger of Android 2.3 JSReftest 1 opt requested. > Retrigger of Android 2.3 JSReftest 2 opt requested. > Retrigger of Android 2.3 Reftest 1 opt requested. > Retrigger of Android 2.3 Reftest 2 opt requested. > Retrigger of Android 2.3 Crashtest 1 opt requested. I accidentally did 2.3 jobs instead of 4.0 jobs. Let's try instead: Retrigger of Android 4.0 Mochitest 1 opt requested. Retrigger of Android 4.0 Robocop 5 opt requested. Retrigger of Android 4.0 Crashtest opt requested. Requesting retrigger of Android 4.0 Reftest 8 opt…

Comment on attachment 8486958 [details] [diff] [review] 1064012.patch Looks good from a mozharness perspective. I assume the buildbot changes will be forthcoming once your verify the tests run as expected on ash?

(In reply to Kim Moir [:kmoir] from comment #7) > Comment on attachment 8486958 [details] [diff] [review] > 1064012.patch > > Looks good from a mozharness perspective. I assume the buildbot changes > will be forthcoming once your verify the tests run as expected on ash? Great! Thanks for the quick review. Similar to what I said on https://bugzilla.mozilla.org/show_bug.cgi?id=1064010#c20, it's not clear why my test suite has to even exist for this to land. I think this should land and start failing (on Cedar only), and then I can use Cedar to build the in-tree bits.

You might want to add this to Ash as well since you can land patches to ash-mozharness w/o reviews. (In case you need to iterate there)

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #9) > You might want to add this to Ash as well since you can land patches to > ash-mozharness w/o reviews. > (In case you need to iterate there) You mean, update Bug 1064010 to enable my test runs on Ash as well as (or instead of) Cedar? That might be sensible. jlund: do you agree?

As well as Cedar.

(In reply to Nick Alexander :nalexander from comment #10) > (In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from > comment #9) > > You might want to add this to Ash as well since you can land patches to > > ash-mozharness w/o reviews. > > (In case you need to iterate there) > > You mean, update Bug 1064010 to enable my test runs on Ash as well as (or > instead of) Cedar? That might be sensible. jlund: do you agree? yup makes sense: both ash and cedar. ash would be nice if you forsee the possible need for many, fast iterations. r+ for https://bug1064010.bugzilla.mozilla.org/attachment.cgi?id=8487506 with `branches_to_keep=['cedar', 'ash']`

Pushed to mozharness/default: https://hg.mozilla.org/build/mozharness/rev/3c712011d313

In prod with reconfig on 2014-09-15 08:30 PT

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.