We're testing a new ad framework for New Tab in Fx69 beta, and we're not seeing any rows in the RedShift tiles database related to this. The pings do arrive at our Pub/Sub as expected. Could you help us understand what is happening, or point us in the right direction?

Last week we recorded the following pings:

TELEMETRY PING: {"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"70.0a1","release_channel":"default","addon_version":"20190729101028","user_prefs":255,"session_id":"n/a","page":"about:newtab","source":"CARDGRID","tiles":[{"id":43165,"pos":0},{"id":43142,"pos":1},{"id":12822678,"pos":2},{"id":43137,"pos":3},{"id":12822660,"pos":4},{"id":43146,"pos":5},{"id":43144,"pos":6},{"id":43140,"pos":7},{"id":43148,"pos":8},{"id":43131,"pos":9},{"id":43124,"pos":10},{"id":43127,"pos":11},{"id":43163,"pos":12},{"id":43133,"pos":13},{"id":43129,"pos":14},{"id":43155,"pos":15},{"id":43135,"pos":16},{"id":43104,"pos":17},{"id":43100,"pos":18},{"id":43080,"pos":19},{"id":43065,"pos":20}],"loaded":21,"action":"activity_stream_impression_stats","impression_id":"{cef47048-eac1-4d1e-a9bc-540288847cfe}","profile_creation_date":17883,"region":"CA"}

TELEMETRY PING: {"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"70.0a1","release_channel":"default","addon_version":"20190729101028","user_prefs":255,"session_id":"n/a","page":"about:newtab","source":"CARDGRID","tiles":[{"id":12822678,"pos":2,"shim":"1,eyJ2IjoiMS4yIiwiYXYiOjQ2ODUxNSwiYXQiOjI0MDEsImJ0IjowLCJjbSI6OTA2MDQyLCJjaCI6MzY4NDgsImNrIjp7fSwiY3IiOjk5MDIyNDMsImRpIjoiNTJiNGIxY2E4MDkzNDIxYjlkMWNmYmU0MGViY2RhZmEiLCJkaiI6MCwiaWkiOiIzMGU2MmZkZjhhN2E0YTM4YTI1ODg1NDRkNGU1MTg3NiIsImRtIjozLCJmYyI6MTI4MjI2NzgsImZsIjo4Njc1ODUyLCJpcCI6IjMuMjE4LjU5LjI2IiwibnciOjEwMjUwLCJwYyI6MCwiZWMiOjAsInByIjoxNDc1MjAsInJ0IjoxLCJycyI6NTAwLCJzYSI6IjgiLCJzYiI6ImktMDYwYzQwM2FkNWZlMWE3MTkiLCJzcCI6OTI4MTAsInN0IjoxMDg0MzY3LCJ1ayI6IntjZWY0NzA0OC1lYWMxLTRkMWUtYTliYy01NDAyODg4NDdjZmV9IiwidHMiOjE1NjQ3NzMxOTQ3MTUsInBuIjoibmV3LXRhYiIsImdjIjp0cnVlLCJncyI6Im5vbmUiLCJiYSI6MSwiZnEiOjB9,KWiwjw6Ctmgqwu-D8USrLE4TYYo"},{"id":43165,"pos":0},{"id":43142,"pos":1}],"action":"activity_stream_impression_stats","impression_id":"{cef47048-eac1-4d1e-a9bc-540288847cfe}","profile_creation_date":17883,"region":"CA"}

Our ad framework has ids in the 12 million range and 12822678 is referenced above. When I try to query for it in RedShift, I don't get any results:

select * from assa_impression_stats_daily
WHERE tile_id > 12000000
limit 10;

0 rows retrieved in 4 s 750 ms

This was caused by one of the ETL jobs for Pocket telemetry in the Tiles data pipeline, which assumed the tile_id should not go over 1e6. Will update that rule for this new change.

Note that this should not affect the GCP based pipeline.

Thanks! I'm glad it can be fixed fairly easily.

It's not affecting the GCP pipeline: both BigQuery and our Pub/Sub are working as expected.

This is now fixed by bug 1572120.