To faciliate fixing bug 1218537 (ingesting multiple job artifacts) while also hopefully helping get rid of datasource (bug 1178641) If we store these objects seperately in their own table it will be easier to update them (as well as fetch them!)

Comment on attachment 8749394 [details] [treeherder] wlach:1270629 > mozilla:master So this is based on the PR I earlier posted to bug 1218537. I have split it into two commits: one that just adds the model to django along with a migration script, and another that modifies the ingestion code. My thinking is that we can then deploy this in two stages: (1) Add the model, convert all existing artifacts (2) Deploy frontend code, flip the switch to ingest into job details instead of saving job info artifacts, migrate any stragglers between the time (1) finished and this step happened. I've done a bunch of local testing, converting all the job info artifacts currently on m-c stage. Everything looks pretty good!

Comment on attachment 8749394 [details] [treeherder] wlach:1270629 > mozilla:master I made a few comments, but nothing terribly serious. Feel free to make the changes or not, as you see fit.

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/d0ea27feb88a44907a4dd2843f6bd1c50a549360 Bug 1270629 - Add initial parts of jobdetail model/API Ingestion not turned on (yet), pending initial migration of data https://github.com/mozilla/treeherder/commit/af9c208792c76c22b0558c1a7c2fc4517f88ef78 Bug 1270629 - Replace Job Info artifacts with a "jobdetail" endpoint This is simpler and removes one more type of artifact from the per-project databases. https://github.com/mozilla/treeherder/commit/238fc614c5b57b29b5d69772d1f08de149bcee42 Bug 1270629 - Temporarily revert changes to remove job info artifact ingestion We'll continue ingesting these artifacts in parallel with the jobdetail objects, to make sure we can revert cleanly if something breaks.

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/7cb214306da114eaed39456772f1b6a8a6040c50 Revert "Bug 1270629 - Temporarily revert changes to remove job info artifact ingestion" This reverts commit 238fc614c5b57b29b5d69772d1f08de149bcee42. https://github.com/mozilla/treeherder/commit/5e8e6d12a3c28f632da237c6758feaed9e9fe03d Revert "Bug 1270629 - Replace Job Info artifacts with a "jobdetail" endpoint" This reverts commit af9c208792c76c22b0558c1a7c2fc4517f88ef78. https://github.com/mozilla/treeherder/commit/635467a77dd91d7b56db5a6fd3eaae3f2d76cf4e Revert "Bug 1270629 - Add initial parts of jobdetail model/API" This reverts commit d0ea27feb88a44907a4dd2843f6bd1c50a549360.

It is not clear to me what changes are required to submit the job info/job details information. Will the old TreeherderJob add_artifact('Job Info', 'json', {'job_details': job_details}) continue to work? What is the preferred method going forward?

(In reply to Bob Clary [:bc:] from comment #7) > It is not clear to me what changes are required to submit the job info/job > details information. Will the old TreeherderJob add_artifact('Job Info', > 'json', {'job_details': job_details}) continue to work? What is the > preferred method going forward? Yes, that will still work (in fact it will still be the only way to submit this information). Eventually we might want to change the artifact format to enable better error checking via jsonschema, but that will happen in a separate bug.

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/645c32fc0daeb04d10c62deb169c039c893e514a Bug 1270629 - Add initial parts of jobdetail model/API Ingestion not turned on (yet), pending initial migration of data https://github.com/mozilla/treeherder/commit/82221633b838bd58224c437876eebbb160a437e8 Bug 1270629 - Replace Job Info artifacts with a "jobdetail" endpoint This is simpler and removes one more type of artifact from the per-project databases. https://github.com/mozilla/treeherder/commit/aa0ceb60285b7f1d20aba85103f73fd6d89fed3d Bug 1270629 - Temporarily revert changes to remove job info artifact ingestion We'll continue ingesting these artifacts in parallel with the jobdetail objects, to make sure we can revert cleanly if something breaks. https://github.com/mozilla/treeherder/commit/2ca0d29aa1e6c79db6ef096e80d657cd4565b84c Merge pull request #1475 from wlach/1270629-2 Bug 1270629 - Add a jobdetail table + endpoint, replacing jobinfo artifacts

Running in production now. Leaving open until I remove the transition/migration code.

Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/3442b2f1193651808a4fa7741d72038abcd7c8d5 Bug 1270629 - Fix broken logviewer

James, are you still using the job info artifacts are can we stop storing those?

Still using them, but I will fix my script when they disappear.

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/477b464dd96eef494ff4942d8fca37ab6fc369b0 Bug 1270629 - Remove migration code for job info -> detail change https://github.com/mozilla/treeherder/commit/e2c92101df5df4486267ac2302f7c72df1bcd1d7 Merge pull request #1529 from wlach/1270629-3 Bug 1270629 - Remove migration code for job info -> detail change

Ok, migration script and legacy job info artifact generation removed. I think we're done here.

Just to clarify - this bug moved a proportion of the rows in the job_artifacts table to a new unified job details table, right? And looking at the now removed migrations script, it seems like the old rows were left in the job_artifacts table (ie copied not moved). So presumably will be covered by the standard cycle_data job as the 4 months window passes? Is this correct? Also, are we ok with the new Heroku stage not having back history for these? (Replication of SCL3 prod -> new Heroku RDS stage stopped running on 11th May, and the now removed migrations script hasn't been run there.)

Correct on all counts. I don't think we care about the job info artifacts on any stage environment (I only ran it on our internal stage for the purposes of verifying that the migration process worked.

Sounds good to me, thank you :-)