We should run update verify tests against beta channel after pushing snippets to the beta channel to test AUS throttling and to prevent bugs like bug 651923.

Example output: bash verify.sh -t -f beta moz192-firefox-linux.cfg --old-updater Using config file moz192-firefox-linux.cfg Using https://aus2.mozilla.org/update/1/Firefox/3.6.16/20110319135224/Linux_x86-gcc3/af/beta/update.xml?force=1 Testing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.17-candidates/build2/update/linux-i686/af/firefox-3.6.16-3.6.17.partial.mar HTTP/1.1 200 OK Server: Apache/2.2.3 (Red Hat) X-Backend-Server: dm-ftp01 Content-Type: application/octet-stream Date: Tue, 26 Apr 2011 09:59:22 GMT Accept-Ranges: bytes Access-Control-Allow-Origin: * ETag: "9c9b0d-cd271-3d776640" Connection: Keep-Alive Last-Modified: Thu, 14 Apr 2011 19:50:41 GMT X-Cache-Info: caching Content-Length: 840305

Not tested yet. It supposed to be run manually after pushing snippets to the beta channel and unthrottling AUS.

* force=1 shouldn't be added to the AUS requests for such tests

Err, wrong file.

* pass "-n" to prevent "force=1" style URLs

I tested the patches in staging. No changes in update verify requests: ================================================ bash verify.sh -c moz20-firefox-mac64.cfg ...... Using config file moz20-firefox-mac64.cfg Using http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Darwin_x86_64-gcc3/af/betatest/update.xml?force=1 05:31:49 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/mac/af/firefox-4.0rc2-4.0.1.partial.mar [2385810/2385810] -> "update/partial.mar" [1] 05:31:53 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/releases/4.0rc2/mac/af/Firefox%204.0%20RC%202.dmg [27893468/27893468] -> "Firefox 4.0 RC 2.dmg" [1] 05:31:56 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/nightly/4.0.1-candidates/build1/mac/af/Firefox%204.0.1.dmg [27904105/27904105] -> "Firefox 4.0.1.dmg" [1] installing downloads/Firefox 4.0 RC 2.dmg ........... FINISH ADD force_plist_reload backup_discard: backup file doesn't exist: force_plist_reload.moz-backup succeeded calling QuitProgressUI Only in source/Firefox.app: force_plist_reload WARN: non-binary files found in diff WARN: check_updates returned warning for Darwin_x86_64-gcc3 downloads/Firefox 4.0 RC 2.dmg vs. downloads/Firefox 4.0.1.dmg: 2 Using http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Darwin_x86_64-gcc3/af/betatest/update.xml?force=1 05:32:40 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/mac/af/firefox-4.0.1.complete.mar [27029123/27029123] -> "update/complete.mar" [1] 05:32:46 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/releases/4.0rc2/mac/af/Firefox%204.0%20RC%202.dmg [27893468/27893468] -> "Firefox 4.0 RC 2.dmg" [1] 05:32:49 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/nightly/4.0.1-candidates/build1/mac/af/Firefox%204.0.1.dmg [27904105/27904105] -> "Firefox 4.0.1.dmg" [1] installing downloads/Firefox 4.0 RC 2.dmg .................. ================================================ Beta update verify output looks like this: Throttled: ================================================ bash verify.sh -t -f beta -n moz20-firefox-linux.cfg .......... Using config file moz20-firefox-linux.cfg Using http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Linux_x86-gcc3/af/beta/update.xml FAIL: no partial update found for http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Linux_x86-gcc3/af/beta/update.xml FAIL: download_mars returned non-zero exit code: 1 ================================================ Unthrottled: ================================================ Using http://staging-stage.build.mozilla.org/update/1/Firefox/3.7a1/20100208064801/Linux_x86-gcc3/en-US/beta/update.xml Testing http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/linux-i686/en-US/firefox-4.0.1.complete.mar HTTP/1.1 200 OK Date: Tue, 26 Apr 2011 12:25:38 GMT Server: Apache/2.0.52 (CentOS) Last-Modified: Thu, 14 Apr 2011 06:20:50 GMT ETag: "5a0003-d41997-ed379480" Accept-Ranges: bytes Content-Length: 13900183 Connection: close Content-Type: text/plain; charset=UTF-8 ================================================ It normally takes 3 minutes to verify beta snippets on darwin10 machines.

Comment on attachment 528301 [details] [diff] [review] buildbotcustom This is a good start, but I think we can do even better. For old branches (3.5/3.6) we need to check that the throttling is disabled for the release channel too. Not clear what we'll need yet for 4.0 going upwards, so we should make the creation of these builders configurable. (In reply to comment #0) > We should run update verify tests against beta channel after pushing snippets > to the beta channel to test AUS throttling and to prevent bugs like bug 651923. Slight correction - ... after pushing snippets to the beta channel and deploying the AUS config change ...

Comment on attachment 528299 [details] [diff] [review] verify.sh changes http://hg.mozilla.org/build/tools/rev/fc636a1a8680

Comment on attachment 528299 [details] [diff] [review] verify.sh changes Ooops, pushed both patches while the second one was r-. Backed out both.

Not working on this bug actively. Back to the pool.

Mass move of bugs to Release Automation component.

Updating summary, as we don't throttle betas these days but do on release. Also, I think it makes sense to run this as part of final_verification since that's now parallelized. QA is currently covering this testing using a script that makes repeated requests on a few urls. A few months ago we added 1 min caching on the Zeus load balancer, so some sort of random query parameter would be necessary if throttling is not 0 or 100%.

Guessing this is still needed?

(In reply to Aki Sasaki [:aki] from comment #13) > Guessing this is still needed? I haven't heard a peep about wanting this since Balrog came along. I think we have much higher confidence in it doing the right thing (vs needing to deploy changes to PHP files in AUS2). I can imagine that we might want something that sanity checks update rate on a continuous basis, but I highly doubt we're ever doing something like this as part of the release process.