Alice is ready for traffic to head her way. You wrote some methods in bug 660331, we should hook those up to send a payload to Alice: - Through a button in the dev tools would be cool but not required. Would need to have some rate limiting - On add-on upload/approval. I'm not sure what is ideal there, I kind of like the idea of approval to cut down on potential automated compromises. CCing jorgev if he has opinions. All of this should be behind a waffle flag, as usual.

(In reply to comment #0) > - On add-on upload/approval. I'm not sure what is ideal there, I kind of > like the idea of approval to cut down on potential automated compromises. > CCing jorgev if he has opinions. I assume you're asking what level of approval the tested add-ons should have. Any file that has received at least Preliminary Approval should be OK to test. Unreviewed add-ons should never be tested like this.

Great, let's trigger it for add-ons when they are approved to prelim or full approval.

Can I get some controlled pushes of test requests into the system so that I can confirm that staging work correctly? We're going to want to work in concert here so that I can shake out bugs on my end.

I think Kumar's last patch let us run these off the CLI - kumar, can you confirm that command? If that's the case, Alice, what data do you want me to send your way?

I'd like to see a test for a linux, macosx and win32 addon. It would also be good to send through some specifically broken requests to ensure that they are handled correctly.

(In reply to comment #4) > I think Kumar's last patch let us run these off the CLI - kumar, can you > confirm that command? ./manage.py perftest --os={os} --firefox={firefox} --file-id={addon-file-id} But the output won't be very useful. I'll fix it up when I start on this for 6.1.9

Wil - with that information would you be able to push through some test runs?

Alice and I played with this a little on IRC. We had great success until her machine disappeared and I haven't talked to her since. Alice, do you know what happened there? Should we hook up our preview server to start sending data to you regularly?

I filed for the timouts we were seeing in bug 679465. It appears to be a known infrastructure issue that IT is already investigating - so it wasn't connected to our testing or code. I will be doing some final work this week to ensure that graph server recognizes the new addon perf testing machines and deals with them correctly. Once that it complete, I would like to get AMO sending data regularly, but it will have to be with the understanding that we will continue to experience timeouts so the system will be *ahem* less than reliable.

Graph server updates are complete. I've hooked the trigger script up to the production master. We are now ready for a Beta launch of the system.

Let's put this behind a waffle flag on AMO (defaulting to off) so we can turn it off if it starts breaking. Also, as with all our remote calls, have a reasonable timeout on it.

Alrighty, we can now start performance tests from the version management page on the AMO devhub, e.g. https://addons.allizom.org/en-US/developers/addon/firebug-kumar-test/versions/ Note that this just hooks up the sender. We don't do anything with the response except log it right now. Alice, does it look like we're sending the right request? Each one of the requests ended in a line like this: ERROR: buildUrl construction: urllib2.HTTPError HTTP Error 404: Not Found 4.0 linux#012 I'm attaching a full log. The lines prefixed with 'DEBUG PERF TEST line' are output received from trigger.cgi. Here is an example of a request we are making from the dev site: http://talos-addon-master1.amotest.scl1.mozilla.com/trigger/trigger.cgi?url=https%3A%2F%2Faddons.allizom.org%2Fdownloads%2Ffile%2F128527%2Ffirebug_kumar_test-1.8.1-Kumar-fx.xpi%3Fsrc%3Dperftest&firefox=firefox4.0&os=macosx64 I'm going to close this bug. We can open new bugs if we need to adjust the way AMO makes trigger requests and also new bugs for displaying results.

Revision to introduce the devhub trigger button https://github.com/jbalogh/zamboni/commit/f436699db2c693905c9851db34a4bed1de1007f0

Looking at where we pull builds from: http://releases.mozilla.org/pub/mozilla.org/firefox/releases/ There is currently only latest-3.6 and latest-6.0 in there - so those are the only versions of firefox that can currently be sent through the system, explaining the error that you seeing.

(In reply to alice nodelman [:alice] [:anode] from comment #14) > There is currently only latest-3.6 and latest-6.0 in there - so those are Fixed: https://github.com/jbalogh/zamboni/commit/828d32b47317e9a8d21ce5ffaccf1c6ba2a71ab7 (I'll kick off another set of tests once I figure out unrelated permission issues on the staging server)

re-opening so I don't lose track of this

(In reply to Kumar McMillan [:kumar] from comment #16) > (In reply to alice nodelman [:alice] [:anode] from comment #14) > > There is currently only latest-3.6 and latest-6.0 in there - so those are > > Fixed: > https://github.com/jbalogh/zamboni/commit/ > 828d32b47317e9a8d21ce5ffaccf1c6ba2a71ab7 I got access to the logs. So, after this change every request to trigger.cgi now times out. Our timeout is set to one second like all other services. Is that too low? Here is an example request: http://talos-addon-master1.amotest.scl1.mozilla.com/trigger/trigger.cgi?url=https%3A%2F%2Faddons-dev.allizom.org%2Fdownloads%2Ffile%2F111865%2Fstatus_watch-0.2.rev11-tb%2Bfx%2Bfn.xpi%3Fsrc%3Dperftest&firefox=firefox4.0&os=win64

Oh, btw, the requests that were timing out were getting sent from the phx data center (which we recently moved to). Does that need to be white listed?

Since the link is up and accessible for me, then yes I would assume that you are outside the whitelist. You should file a bug with IT. They maintain a mana page detailing the network flow for the addon master vm: https://mana.mozilla.org/wiki/display/websites/talos-addon-master1.amotest.scl1.mozilla.com#

Hi. AMO in PHX got whitelisted and we are able to connect to Talos and start the performance tests. Yay. Alice, I'm still seeing some errors in the service response from a test run I did on 2011-09-07. Is there anything that we need to change on the AMO side to correct this? I'll cc the log.

Lines prefixed with 'DEBUG PERF TEST line' were what the server responded with. I see some URLError timeouts (in AMO) and also 'ERROR: pingUrl, server error: 404' in Talos.

I pushed through an example link from your log: http://talos-addon-master1.amotest.scl1.mozilla.com/trigger/trigger.cgi?url=https%3A%2F%2Faddons-dev.allizom.org%2Fdownloads%2Ffile%2F129535%2Ffirebug_kumar_test-1.8.1-Kumar-fx.xpi%3Fsrc%3Dperftest&firefox=firefox6.0&os=win64 And it completed with: "SENDCHANGE: change sent successfully"

Maybe you should try just hitting the links from the command line to see if you can isolate something in your script that is interfering? Because the links seem good and working on my end.

Also, please limit your testing to 'macosx64', 'win32' and 'linux'. These are the only platforms that we have test slaves for. It you attempt to test other platforms the sendchanges are simply ignored.

We should file a "have your way with" bug for this with steps on how to test it so QA isn't digging through this bug

I've updated https://wiki.mozilla.org/Buildbot/Talos/AddonTrigger#Variables It now indicates the limitation as to supported platforms and the limitation on the versions of firefox that can be tested.

Here is a bug to explore and test this further: bug 689187 I'm closing this one since we have an initial connection working. Thanks for your help Alice.