To facilitate moving more project-specific artifacts to the master database, it would be useful to create a table/model corresponding to the job. This way we could track guid, id, in one place, and look it up easily. Eventually, this table could be fleshed out into a full-fledged replacement for the project-specific jobs table.

Comment on attachment 8741901 [details] [treeherder] wlach:1265037 > mozilla:master Hi James, I think you'd be a good person to take a look at this. This isn't very useful by itself, but I think it should make the eventual migration away from datasource easier, while we incrementally migrate away from it with things like bug 1218537.

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/104787282d0db965ea948342ce1f2926936f3d27 Bug 1265037 - Create an intermediate jobs model/table Not really useful in-and-of-itself, but should be helpful by acting as a centralized place to relate and map job information from what is stored in the main treeherder database to what's in the per-project databases. Eventually, we can extend this into a full-fledged jobs model. https://github.com/mozilla/treeherder/commit/081d86bd7b699af7128841c266939464cdd795c9 Bug 1265037 - Temporarily disable creation of intermediate job entries We will wait to turn this on until the initial migration is complete to ensure jobs are more-or-less in chronological order https://github.com/mozilla/treeherder/commit/4a49dfe6938a43980c24c7d8ecb116918e0d44a3 Merge pull request #1411 from wlach/1265037 Bug 1265037 - Create an intermediate jobs model/table

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/f744b91de1b82a35c018c64ca09b17abfbd3fbe8 Revert "Bug 1265037 - Temporarily disable creation of intermediate job entries" This reverts commit 081d86bd7b699af7128841c266939464cdd795c9. https://github.com/mozilla/treeherder/commit/43ff9980498ff1d179bbff1e30aa83cac35ed226 Bug 1265037 - Adapt migration script to assume most jobs migrated https://github.com/mozilla/treeherder/commit/ceaa8d1b629b0ef4767cb5a7eb1f2ec1e208ac23 Merge pull request #1425 from wlach/1265037-2 Bug 1265037 - Enable dynamic intermediate job ingestion

Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/78e7d06233db8b71f233d0ccfb9bfec3ba0a32af Bug 1265037 - Fixes to the final migration script

Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/a36773fcf4135291d50605370e3650e2df415b6c Bug 1265037 - Fix edge case in migration script If ingestion was ongoing during the migration, it was possible for things to error out since operations were batched inside a transaction. Since it should be rare to need to insert new items when this script is being run, just skip the transaction.

Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/363a90a53eb806ce8241df81fae35c08154bb6e5 Bug 1265037 - Fix linter error

The migration was horrible and required hand-hacking, but I believe this is now done.