We want to move from uploading Breakpad symbols using SCP to using Socorro's upload API. We'll need an auth key to do this, though, which will need to live on the build machines.

coop: I think we want to handle this similar to the Google API key--a puppet'ed secret that winds up in a known location on the build machines so we can pass it to a configure option in mozconfigs.

Someone pointed me at https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Add_new_secrets on #releng yesterday, FWIW.

As an update, I've added the token to hiera (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets#Secrets) and will roll the puppet patches to deploy this to build machines tomorrow.

(In reply to Chris Cooper [:coop] from comment #3) > As an update, I've added the token to hiera > (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets#Secrets) > and will roll the puppet patches to deploy this to build machines tomorrow. Flight delay in your favor, collect patch now. Token will live in C:/builds/crash-stats-api.token (Windows) or /builds/crash-stats-api.token (POSIX). This is based on the patch for the google oauth key.

Comment on attachment 8570278 [details] [diff] [review] [puppet] Add API token for crash-stats upload Review of attachment 8570278 [details] [diff] [review]: ----------------------------------------------------------------- https://hg.mozilla.org/build/puppet/rev/a8af1412e229

I've merged to the production tag, and verified that builders receive the new token once they re-puppetize. We'll need to wait a day or two for the golden AMIs to pick up this change, but we should be ready to start using the token by Monday.

Should this be ready to go by now?

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7) > Should this be ready to go by now? For Linux and Mac, yes. I've double-checked that the token is present there. The Windows builders still need GPO created to deploy the file.

(In reply to Chris Cooper [:coop] from comment #8) > The Windows builders still need GPO created to deploy the file. Filed bug 1139649.

Seems fixed now, thanks!

Apparently this didn't get completely fixed, the nightly build on bld-linux64-spot-128 didn't have the token file: https://treeherder.mozilla.org/logviewer.html#?job_id=1197057&repo=mozilla-central 08:51:15 INFO - /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/obj-firefox/_virtualenv/bin/python /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-39.0a1.en-US.linux-i686.crashreporter-symbols-full.zip' 08:51:15 INFO - Error: SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist! 08:51:15 INFO - gmake[1]: *** [uploadsymbols] Error 1 Same error on the 32-bit nightly on bld-linux64-spot-1036: https://treeherder.mozilla.org/logviewer.html#?job_id=1197058&repo=mozilla-central

The token is there, but the permissions are set to 0600, same as the google-oauth-api.key. I'll switch this to 0644 in puppet.

Except it's failing here: https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py#45 Which indicates that the script can't see the file at all. 0600 should be fine as long as the owner of the file is the user we run builds as.

ahhh of course, we need to copy it into the mock environment, ala: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#261

Comment on attachment 8581696 [details] [diff] [review] [buildbot-configs] Copy crash-stats token into mock env Review of attachment 8581696 [details] [diff] [review]: ----------------------------------------------------------------- stamp

Based on IRC chatter and confusion on if this was a flow issue, I edited "firewall-tests" and added a test to ensure this flow remains open: To ssh://gitolite3@git-internal.mozilla.org/releng/firewall-tests d40176b..60a81c1 master -> master $ git diff -U0 d40176b..60a81c1 diff --git a/test_buildslaves.py b/test_buildslaves.py index 9697738..ec7e9f3 100644 --- a/test_buildslaves.py +++ b/test_buildslaves.py @@ -44,0 +45,5 @@ slaveapi = IPSet([ +crashstats = IPSet([ + host('crash-stats.mozilla.org'), + host('crash-stats.allizom.org') +]) + @@ -69,0 +75,8 @@ def test_blobupload_access(): + +def test_crashstats_access(): + """All buildslaves can access crashstats""" + # XXX Only hosts that compile code need the access + rules.assertPermits( + slave_vlans, + crashstats, + 'https')

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5d5b09f943ea

Presumably this should be fixed now. Thanks coop for the patch and Callek for figuring out the issue! Maybe one of you can edit the docs at https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Add_new_secrets so this extra step is noted for anyone adding a similar secret file for the build in the future?

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #19) > Maybe one of you can edit the docs at > https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/ > Add_new_secrets so this extra step is noted for anyone adding a similar > secret file for the build in the future? Done: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Add_new_secrets

This is somehow still broken because I'm not allowed to have nice things: https://treeherder.mozilla.org/logviewer.html#?job_id=1218661&repo=mozilla-central https://treeherder.mozilla.org/logviewer.html#?job_id=1218662&repo=mozilla-central

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #21) > This is somehow still broken because I'm not allowed to have nice things: > https://treeherder.mozilla.org/logviewer.html#?job_id=1218661&repo=mozilla- > central > https://treeherder.mozilla.org/logviewer.html#?job_id=1218662&repo=mozilla- > central On linux, no less. I'll dig into this today.

The mozharness script, fx_desktop_build.py, is getting its list of copyin files from the mozharness config files rather than buildbot: https://hg.mozilla.org/build/mozharness/file/e1db791a6e6f/configs/builds/releng_base_linux_32_builds.py#l37 https://hg.mozilla.org/build/mozharness/file/e1db791a6e6f/configs/builds/releng_base_linux_64_builds.py#l37 Patch incoming.

Comment on attachment 8581696 [details] [diff] [review] [buildbot-configs] Copy crash-stats token into mock env Review of attachment 8581696 [details] [diff] [review]: ----------------------------------------------------------------- https://hg.mozilla.org/build/buildbot-configs/rev/5d5b09f943ea

Comment on attachment 8587035 [details] [diff] [review] [mozharness] Copy crash-stats token into mock env Review of attachment 8587035 [details] [diff] [review]: ----------------------------------------------------------------- looks good. Do we need to do the same for the b2g desktop builds? http://mxr.mozilla.org/build/source/mozharness/configs/b2g/desktop_linux64.py#35 http://mxr.mozilla.org/build/source/mozharness/configs/b2g/desktop_linux32.py#35

Depends on whether those are going to wind up in taskcluster, but yes, everything that's uploading symbols currently is going to need this treatment.

Comment on attachment 8587035 [details] [diff] [review] [mozharness] Copy crash-stats token into mock env Review of attachment 8587035 [details] [diff] [review]: ----------------------------------------------------------------- Updated the b2g desktop files as well: https://hg.mozilla.org/build/mozharness/rev/7c12014639c0

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Okay, so this just needs the in-tree pinned mozharness rev bumped to pick this up now?

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #30) > Okay, so this just needs the in-tree pinned mozharness rev bumped to pick > this up now? Yes, and you can do this yourself, or ask jlund to do it for you.

Someone else updated mozharness.json, my procrastination paid off!