The token server no longer appears to be returning the hashed_fxa_uid property in it's response, as of around Oct 20th. The sync ping code adds this to the ping data without looking at it (meaning it's missing), and so this causes the Scala code that inserts the ping into redash to ignore these pings, since it considers this to be a required field.

This deploy went through at about that time (Oct 19th): https://bugzilla.mozilla.org/show_bug.cgi?id=1310911#c3 But it's just an update to the browserid-verifier library, not to any of the tokenserver code. I guess it's possible that a bug in this update has stopped it from handing back the extra idpClaims in the FxA assertion, I'll see if I can repro.

Benson, can you please confirm that git shas of tokenserver and browserid-verifier are running in production?

I don't have an explanation for this on the code side, I'm wondering if we e.g. accidentally deployed an older version of tokenserver when we did the browserid-verifier update above. Unfortunately tokenserver doesn't provide the nice /__version__ endpoint that would tell us what version it's running.

Production is running 1.2.23 which is the latest version.

Per conversation with :mostlygeek in IRC, let's roll back to the previous tokenserver stack and see whether that fixes the problem. If so, we can then take a look at the new stack and try to figure out WTF is going on.

Rolled back to previous stack. Versions: - Tokenserver: 1.2.23 - browserid-verifier: 0.6.0

This appears to have fixed the issue, tokenserver is now returning device-ids again. I've got some deep dive debugging in my future to figure out what went wrong here...

I haven't been able to reproduce this locally, I think we may have to actually debug the live and buggy stack. :mostlygeek, I'm attaching a text file containing a harmless test assertion. Could you please see if the local verifier running on the new tokenserver stack will verify it, and if so what it produces? For comparison, this is what the hosted FxA verifier has to say: $ curl --header 'Content-Type: application/json' -d @TEST_ASSERTION.txt https://verifier.accounts.firefox.com/v2 {"audience":"https://example.com","expires":1509169527118,"issuer":"mockmyid.s3-us-west-2.amazonaws.com","email":"tester@mockmyid.s3-us-west-2.amazonaws.com","idpClaims":{"fxa-deviceId":"12345678"},"status":"okay"} We need to make sure the local verifier instance is properly returning the "idpClaims" hash like in the above output.

Here's the output on the local verifier: {"audience":"https://example.com","expires":1509169527118,"issuer":"mockmyid.s3-us-west-2.amazonaws.com","email":"tester@mockmyid.s3-us-west-2.amazonaws.com","idpClaims":{"fxa-deviceId":"12345678"},"status":"okay"}

> "idpClaims":{"fxa-deviceId":"12345678"} OK, so that looks fine.

I'm really at a loss to explain this, as the code for returning `hashed_fxa_uid` doesn't take any conditional branches or anything. :mostlygeek, a couple more questions for you: * While it's out of proudction, is the broken stack HTTP-accessible anywhere that I might be able to run further tests against it? * Can you find the file `tokenserver/views.py` and just verify that it is indeed trying to return the hased_fxa_uid field, like it is here: https://github.com/mozilla-services/tokenserver/blob/1.2.23/tokenserver/views.py#L306

I dug deeper and it's my fault. I migrated tokenserver to our new jenkins pipeline and had a single byte bug that caused the wrong version of tokenserver to be deployed. I'm deploying it to stage and will verify that the issue is resolved. In comment #4 I only checked that the correct version of TokenServer was passed to the cloudformation scripts.

Deployed to production and I ssh'd into the servers and verified that fxa_uid and device_id are part of the app's log output.