The below push shows on TBPL (with running jobs). Treeherder never picked it up. https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=f41dc09ee34b The push after (https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=a70c83745785) appears on both as normal.

I noticed it shows on treeherder-dev, so there is nothing wrong with that push specifically. I guess for some reason the pushlog cache is striking again. I'm proposing to stop caching the last pushId fetched from each repo, since it's probably giving us more pain than benefits. :jeads :camd do you have an opinion on that?

What makes you think it's the pushlog ID cache? I've just cleared the cache and nothing appeared at: https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=f41dc09ee34b Surely the issue is more "ingestion job fails/is raced -> we then never attempt to ingest that push ID again, even if we're using the full 24 hours of pushlog".

the reason why it's not ingesting that push even after the cache was cleared is that the maximum number of results returned by the pushlog service is 10 (I just realized it) and that push is at the moment 11th. I think the best solution is to remove the pushlog id cache and increase the number of elements fetched.

Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/ba881e203092d6f016725ddf92eefb810db577cd (Bug 1071577) - remove pushlog last fetched cache; also increase the pushlog time range to the last 24hr https://github.com/mozilla/treeherder-service/commit/24c437d0d6d5950386489dd1852c1585f83adc93 Merge pull request #227 from mozilla/bug-1071577-remove-pushlog-cache (Bug 1071577) - remove pushlog last fetched cache; also increase the pus...

(In reply to Mauro Doglio [:mdoglio] from comment #3) > the reason why it's not ingesting that push even after the cache was > cleared is that the maximum number of results returned by the pushlog > service is 10 (I just realized it) and that push is at the moment 11th. I > think the best solution is to remove the pushlog id cache and increase the > number of elements fetched. The commit that landed here added &maxhours=24 to the json-pushes request URL, which is what I presume you meant by "increase the number of elements fetched"? If so, I don't think that will help, since "maxhours" isn't a valid json-pushes param - I know TBPL passes it, but there's no reference to it in the pushlog repo: https://hg.mozilla.org/hgcustom/pushlog/file/86e6470a7c55/pushlog-feed.py#l255 As such, I believe the landing here actually makes things worse, since if there were > 10 pushes since the last request (I guess unlikely, but still), then we'll actually lose them forever, which we wouldn't have done when requesting by cached push ID directly.

Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/d7908793a59c410b0418c28cce712376af4e11d8 Bug 1071577 - Cache revision hashes to reduce pushlog load https://github.com/mozilla/treeherder-service/commit/2c258f443d8c2f65838886614d5f24d5d78a11f9 Bug 1071577 - Moved result_set revision hash cache step to after associated revision storage https://github.com/mozilla/treeherder-service/commit/4f7b069a34d7ef94bc7ae588a725846e8c2083db Merge pull request #228 from mozilla/bug-1071577-revision-hash-cache Bug 1071577 revision hash cache

Changing summary to reflect what work was carried out; part 2 is occurring in bug 1072291.