Seems to only be on AMO though. myfavoritebeer.org (and others) are quick. STR: 1) Click log in with browser id 2) click log in in the popup 3) popup goes away, AMO just sits there for ~10s 4) AMO loads new page, user is logged in

99% sure this is our fault and not browserid's. I put a graphite decorator on the browserid_login function meaning it timed the entire thing. Due to some bugs in how the library works with the threading in mod_wsgi that started failing after a few minutes but not until I got several data points. I moved the timing to measure just the auth.authenticate() function (which is all in the browserid library) and it was very fast. Therefore, I'm proposing this is on our end and somewhere in the _login() function. Screenshot attached.

Yeah its in auth.login. Logging in without browser id is just as slow and getting to the 400 when trying to login as an "special" user is fast too.

MySQL struggling on this box, probably due to the cron jobs. Specifically oremj noted update_addons_collections_downloads, turning off celery sped up browserid a lot. We should turn some of these cron jobs off on -dev if they aren't relevant, eg: metrics update scripts.

Confirmation that I timed this over the network to eliminate the various magical middlewares on both sides and have yet to see a request/response cycle take > 30ms from browserid.org (at least on a verify flow). In particular the JSON replies for /wsapi/list_emails and /verify? go back to addons-dev and I don't see the next HTTP request from addons-dev for 5 to 6 seconds on the browserid.org side. My takeaway is that we could use some automated tests on the browserID side of the house to test transaction latency when in doubt.

We've improved -dev a bit by removing some unnecessary cron jobs. There's nothing browserID specific here, it's just a slow -dev. We'll retest when we get it onto production hardware. Won't fix, but will reopen if there's a problem on prod.