Many of our problems go away if we move the pageloader extension into Talos. To the best of our knowledge, nothing uses pageloader outside of talos, and if anyone needs the XPI, we can generate that. Talos will have to be fixed such that it can us the unpacked pageloader extension. That is about it, though I'm sure there's a bit of cleanup in a few places (there always is). I'm not sure what needs to change on the releng side of things. Maybe nothing initially. :armen?

deprecates https://bugzilla.mozilla.org/show_bug.cgi?id=720526

We should decide if we care about history or not. Obviously if we don't it is easier

IRC conversation with jmaher seems to indicate we want to keep history. http://mercurial.selenic.com/wiki/MergingUnrelatedRepositories indicates how we might do this. I'm not sure how we can review this. Since we're merging history there is no patch. Should we just....do this and be careful about it?

If pageloader will come through talos.zip rather than downloading it directly we would have to do the following changes: 1) stop downloading pageloader.xpi 2) when unzipping talos.zip we should put the pageloader.xpi under "talos/page_load_test" like it currently does How important is this? Would it be better to do it when we have switched to mozharness rather than doing this twice? Changing this would also require adding a flag to customize which branches do the old way and which the new way. I would hate making the factory yet more complicated so deferring this would be my preferred way. Makes sense?

So we need to do something now to make Talos usable for jetperf, and adding pageloader is the easiest option there. Although I'm open to ideas. That said, we don't necessarily need to move the automation side to use the pageloader in talos at the same time, if we want to do it in two separate stages

We'll also want to add fennecmark to talos, assuming there is no conflict there: http://hg.mozilla.org/users/tglek_mozilla.com/fennecmark https://wiki.mozilla.org/Mobile/Fennec/Android#talos

There are also *two* copies of pageloader in mozilla-central: http://mxr.mozilla.org/mozilla-central/source/layout/tools/pageloader/ http://mxr.mozilla.org/mozilla-central/source/testing/tools/pageloader/ These should be eliminated too.

Example merge: http://k0s.org/mozilla/hg/talos via hg-merge.py --master http://hg.mozilla.org/build/talos http://hg.mozilla.org/build/pageloader#talos/pageloader (http://k0s.org/hg/config/file/tip/python/hg-merge.py)

This wires in the in-tree pageloader to talos. Tested locally. You will need to perform the command in comment 8 or otherwise have talos/pageloader in your tree. (I can also go ahead and push talos/pageloader if desired; it shouldn't cause any problems). This doesn't solve the mobile problem and I would like to talk about how to do that. Ideally, I would like to do so the same as desktop, but see bug 727177 about this. mobile installs pageloader as an extension, not as a bundle. We could do a few things: 1. make pageloader an extension for desktop and get rid of the bundles concept. we don't use bundles on mobile and it would be nice to be as consistent as possible between the two cases 2. have a default --extensions for mobile *ONLY* (see 1.) of ${talos}/pageloader; of course, how would this work with the command line? :shrug: If you have it in the yml config file and then use --extension, you will overwrite the value....unless you force the default to be ['${talos}/pageloader'] in remoteTalosOptions (http://hg.mozilla.org/build/talos/file/ff646574951c/talos/remotePerfConfigurator.py#l152) 3. have run_tests.py/ffsetup.py/whatever.py add the path to pageloader to extensions iff it is not there already. Again, for mobile *ONLY* Personally I prefer option 1 assuming that stakeholders are satisfied. It would be nice to A. have everything work the same way; B. delete code and complexity

Comment on attachment 597210 [details] [diff] [review] fix desktop case Review of attachment 597210 [details] [diff] [review]: ----------------------------------------------------------------- a few questions and more editing of every single .config file! ::: talos/addon.config @@ +63,5 @@ > noscript.default: 'file:// about:blank about:credits addons.mozilla.org mozilla.net flashgot.net google.com gstatic.com googleapis.com securecode.com informaction.com yahoo.com yimg.com yahooapis.com maone.net noscript.net hotmail.com msn.com passport.com passport.net passportimages.com live.com js.wlxrs.com' > network.dns.ipv4OnlyDomains: localhost > > > +# Extensions to install in test (use "extensions: []" for none) what would an example of an extension be? Likewise do we even use this anymore? ::: talos/ffsetup.py @@ +76,5 @@ > self._deviceroot = options['deviceroot'] > self._host = options['host'] > self._port = options['port'] > self._env = options['env'] > + self._hostproc = hostproc or self.ffprocess python 2.4 compatible? ::: talos/run_tests.py @@ +505,5 @@ > + # https://bugzilla.mozilla.org/show_bug.cgi?id=705809 > + def interpolatePath(path): > + return string.Template(path).safe_substitute(talos=here) > + browser_config['bundles'] = dict([(i, interpolatePath(j)) > + for i,j in browser_config['bundles'].items()]) we only do this in bundles? @@ +558,5 @@ > port = url.port > > if port: > import mozhttpd > + httpd = mozhttpd.MozHttpd(host=url.hostname, port=int(port), docroot=here) shouldn't docroot=os.path.split(here)[] ?

(In reply to Joel Maher (:jmaher) from comment #10) > Comment on attachment 597210 [details] [diff] [review] > fix desktop case > > Review of attachment 597210 [details] [diff] [review]: > ----------------------------------------------------------------- > > a few questions and more editing of every single .config file! > > ::: talos/addon.config > @@ +63,5 @@ > > noscript.default: 'file:// about:blank about:credits addons.mozilla.org mozilla.net flashgot.net google.com gstatic.com googleapis.com securecode.com informaction.com yahoo.com yimg.com yahooapis.com maone.net noscript.net hotmail.com msn.com passport.com passport.net passportimages.com live.com js.wlxrs.com' > > network.dns.ipv4OnlyDomains: localhost > > > > > > +# Extensions to install in test (use "extensions: []" for none) > > what would an example of an extension be? Likewise do we even use this > anymore? I use this for JetPerf -- in fact, jetperf relies on this. extensions would be a path to an xpi or directory. Before my time, apparently, they were a dict of (ID, path), but before I started working on talos it was just a list. The text was just wrong. > ::: talos/ffsetup.py > @@ +76,5 @@ > > self._deviceroot = options['deviceroot'] > > self._host = options['host'] > > self._port = options['port'] > > self._env = options['env'] > > + self._hostproc = hostproc or self.ffprocess > > python 2.4 compatible? Of course. python 1.4 compatible. > ::: talos/run_tests.py > @@ +505,5 @@ > > + # https://bugzilla.mozilla.org/show_bug.cgi?id=705809 > > + def interpolatePath(path): > > + return string.Template(path).safe_substitute(talos=here) > > + browser_config['bundles'] = dict([(i, interpolatePath(j)) > > + for i,j in browser_config['bundles'].items()]) > > we only do this in bundles? Currently. I can add the other things as part of this patch if desired, though there is also https://bugzilla.mozilla.org/show_bug.cgi?id=705809 for it. I would like to figure out what to do about the mobile case before being too extensive here. > @@ +558,5 @@ > > port = url.port > > > > if port: > > import mozhttpd > > + httpd = mozhttpd.MozHttpd(host=url.hostname, port=int(port), docroot=here) > > shouldn't docroot=os.path.split(here)[] ? The current code is http://hg.mozilla.org/build/talos/file/ff646574951c/talos/run_tests.py#l549 docroot=os.path.split(os.path.realpath(__file__))[0] which `here` should be equivalent to. (In any case, this does work with tsvg).

(In reply to Jeff Hammel [:jhammel] from comment #9) > Created attachment 597210 [details] [diff] [review] > fix desktop case > > This wires in the in-tree pageloader to talos. Tested locally. You will > need to perform the command in comment 8 or otherwise have talos/pageloader > in your tree. (I can also go ahead and push talos/pageloader if desired; it > shouldn't cause any problems). > > This doesn't solve the mobile problem and I would like to talk about how to > do that. Ideally, I would like to do so the same as desktop, but see bug > 727177 about this. mobile installs pageloader as an extension, not as a > bundle. We could do a few things: > > 1. make pageloader an extension for desktop and get rid of the bundles > concept. we don't use bundles on mobile and it would be nice to be as > consistent as possible between the two cases > > 2. have a default --extensions for mobile *ONLY* (see 1.) of > ${talos}/pageloader; of course, how would this work with the command line? > :shrug: If you have it in the yml config file and then use --extension, you > will overwrite the value....unless you force the default to be > ['${talos}/pageloader'] in remoteTalosOptions > (http://hg.mozilla.org/build/talos/file/ff646574951c/talos/ > remotePerfConfigurator.py#l152) > > 3. have run_tests.py/ffsetup.py/whatever.py add the path to pageloader to > extensions iff it is not there already. Again, for mobile *ONLY* > > Personally I prefer option 1 assuming that stakeholders are satisfied. It > would be nice to A. have everything work the same way; B. delete code and > complexity Decided to go with option 2

You will need to have pageloader and fennecmark in the appropriate places to use this patch.

Tested locally on android with ts an tsvg

Running on try: https://tbpl.mozilla.org/?tree=Try&rev=de8c443a767e

Comment on attachment 597480 [details] [diff] [review] wire up for desktop and mobile Review of attachment 597480 [details] [diff] [review]: ----------------------------------------------------------------- ::: talos/run_tests.py @@ +568,5 @@ > # run the tests > results = {} > utils.startTimer() > utils.stamped_msg(title, "Started") > + csv_filters = [filter.ignore_max, filter.mean] this isn't related to pageloader. @@ +616,5 @@ > utils.stamped_msg("Failed sending results", "Stopped") > #failed to send results, just print to screen and then report graph server error > for test in tests: > testname = test['name'] > + send_to_csv(None, {testname : results[testname]}, csv_filters) this isn't related to pageloader

(In reply to Joel Maher (:jmaher) from comment #16) > Comment on attachment 597480 [details] [diff] [review] > wire up for desktop and mobile > > Review of attachment 597480 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: talos/run_tests.py > @@ +568,5 @@ > > # run the tests > > results = {} > > utils.startTimer() > > utils.stamped_msg(title, "Started") > > + csv_filters = [filter.ignore_max, filter.mean] > > this isn't related to pageloader. > > @@ +616,5 @@ > > utils.stamped_msg("Failed sending results", "Stopped") > > #failed to send results, just print to screen and then report graph server error > > for test in tests: > > testname = test['name'] > > + send_to_csv(None, {testname : results[testname]}, csv_filters) > > this isn't related to pageloader No they aren't. Currently the latter items is broken due to my mistake landing http://hg.mozilla.org/build/talos/rev/5d955efe4678 and not passing in filters there. So we can leave it broken and file a follow up bug, reopen bug 723569 and set it as a blocker here for the follow-up patch, reopen bug 723569 back out the whole patch, unbitrot, test and reland, or just fix here. I only found this out as I did not (as unusual) set a --result_url and remote.config, unlike other config files, does: http://hg.mozilla.org/build/talos/file/ff646574951c/talos/remote.config#l10 . So following the successful ts run, I got an 403 error when trying to post to http://graphs-stage.mozilla.org/server/collect.cgi . Our fallback for posting errors -- strangely -- is to write csv to screen.

Command from talos hg directory: hg-merge.py http://hg.mozilla.org/build/pageloader#talos/pageloader http://hg.mozilla.org/users/tglek_mozilla.com/fennecmark#talos/mobile_profile/extensions/bench@taras.glek

dev-platform: http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/2bc1da3afa7f7153#

(In reply to Jeff Hammel [:jhammel] from comment #15) > Running on try: https://tbpl.mozilla.org/?tree=Try&rev=de8c443a767e A minor windows bug found, retry is pretty green: https://tbpl.mozilla.org/?tree=Try&rev=716d448271d4

pushed: http://hg.mozilla.org/build/talos/rev/56020a3a3bba

Oops, was a bit premature closing. Next steps: removing hg.m.o/build/pageloader and removing the two instances in m-c . Also, should mail taras and see if he wants to remove his fennecmark repo

Will also need to remove the pageloader.xpi download step for buildbot

We can probably get rid of standalone talos too: bug 714659

Calling this done. Reopen with comments if there is any reason to keep active