Closed
Bug 1281820
Opened 8 years ago
Closed 8 years ago
Stop using/storing bug suggestion artifacts, just generate them dynamically
Categories
(Tree Management :: Treeherder, defect)
Tree Management
Treeherder
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: wlach, Assigned: wlach)
References
Details
Attachments
(3 files, 1 obsolete file)
The next logical step after bug 1266062 is to migrate the error summary artifacts into the Django ORM. Lots of potential for cleanup and simplification here, as well as hopefully reducing the amount of redundant data.
Assignee | ||
Comment 1•8 years ago
|
||
After some experimentation, I believe we should be able to just generate bug suggestions dynamically, without bothering to store them. This should be beneficial from a lot of angles (ensuring they're relatively up to date, ease of modification, disk space usage...)
Summary: Migrate error summary into Django ORM → Stop using/storing bug suggestion artifacts, just generate them dynamically
Comment 2•8 years ago
|
||
Assignee | ||
Comment 3•8 years ago
|
||
Comment on attachment 8793921 [details]
[treeherder] wlach:1281820 > mozilla:master
Hey James, most of the changes here were related to autoclassification in one way or another, so I'm passing this to you. I (tried to) clean up the tests as well as part of this patch, which have seperate commits, but I think this is best reviewed as a unit.
Attachment #8793921 -
Flags: review?(james)
Comment 4•8 years ago
|
||
The more I think about this, the more I think it's an awesome idea.
Traditionally Treeherder has had a very write-heavy DB workload (which hinders with many things - replication included), and this is one step to making it more read-heavy (well, with memcached, not much so, but same idea). It's much easier to scale out read-capacity, than write.
Updated•8 years ago
|
Attachment #8793921 -
Flags: review?(james) → review+
Assignee | ||
Comment 5•8 years ago
|
||
It probably doesn't make sense to land this when (1) we're in the middle of the heroku migration and (2) I'm about to go on PTO (until next Wednesday). I'll do that when I return next week.
Comment 6•8 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/69bdc2ed155226c8760b1ca23c70345d85e70ce8
Bug 1281820 - Clean up autoclassification unit tests
* Remove unused fixtures
* Add test_job, test_job_2 fixtures for testing job data, instead of
ingesting 11 jobs
* Remove various uses of datasource
https://github.com/mozilla/treeherder/commit/9b9d37e6903589824eaa5456489defb56784ff57
Bug 1281820 - Move autoclassify methods into Django ORM class
https://github.com/mozilla/treeherder/commit/8fc9754f4e0665cd0fd18530ed4196868b3179e8
Bug 1281820 - Stop using bug suggestions artifacts
Instead, generate the data when required. We will store the return value
of this in memcache for a day to ensure things are responsive for the sheriffs
when classifying recent failures.
I'm getting 404 errors when selecting jobs on ./web-server.js with a recent pull from master:
https://treeherder.mozilla.org/api/project/mozilla-inbound/jobs/37297801/bug_suggestions/
HTTP/1.1 404 NOT FOUND
Connection: keep-alive
Server: gunicorn/19.6.0
Date: Sat, 08 Oct 2016 05:26:16 GMT
Transfer-Encoding: chunked
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Origin: *
x-frame-options: DENY
X-Content-Type-Options: nosniff
Via: 1.1 vegur
Flags: needinfo?(wlachance)
Assignee | ||
Comment 8•8 years ago
|
||
(In reply to Wes Kocher (:KWierso) from comment #7)
> I'm getting 404 errors when selecting jobs on ./web-server.js with a recent
> pull from master:
Yeah, the server side changes are not yet deployed to production (I wanted to let this large change soak on stage for the weekend).
Meanwhile you can run `./web-server.js --stage` and the UI will work against stage, which does have these changes.
Flags: needinfo?(wlachance)
Comment 9•8 years ago
|
||
Assignee | ||
Updated•8 years ago
|
Attachment #8801322 -
Flags: review?(james)
Assignee | ||
Comment 10•8 years ago
|
||
Just posted a followup PR to actually remove the bug suggestion artifact storage code, now that we've validated things are working ok in production. Once that's applied, I think we can resolve this bug.
Comment 11•8 years ago
|
||
Assignee | ||
Updated•8 years ago
|
Attachment #8801723 -
Attachment is obsolete: true
Updated•8 years ago
|
Attachment #8801322 -
Flags: review?(james) → review+
Comment 12•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/b035647fe2543a1b657d66bb0b91a7d2e321fe83
Bug 1281820 - Remove all bug suggestion artifact storage code + tests (#1927)
Comment 13•8 years ago
|
||
Comment 14•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/5abb9e86088c364a8d394b6c18665ec54e9446ca
Bug 1281820 - Revert db schema changes temporarily (#1933)
They need to be applied seperately
Comment 15•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/1b0d0f27f8a258ef825150790783c31e24cd5fef
Revert "Bug 1281820 - Revert db schema changes temporarily (#1933)"
This reverts commit 5abb9e86088c364a8d394b6c18665ec54e9446ca.
Comment 16•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/6ff8cef53273dfc95d85c4f8b21a25e88bb24f61
Bug 1281820 - Remove a few more references to artifacts from the ui
Comment 17•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/5cef814f0f7670c12bc847d04a361aea4e8a5f13
Bug 1281820 - Revert db schema changes temporarily (again)
These are not being applied (there are complaints about integrity
errors because of job guid conflicts), I think there are some issues with
dropping columns inside an actively updated table. Reverting until we
can figure them out.
Assignee | ||
Comment 18•8 years ago
|
||
I am having trouble getting the column deletes for the text log summary table applying, so have reverted them for now. We seem to be getting integrity errors during the migration process, probably due to some mysql quirk. Here's what I'm seeing in the papertrail logs:
Oct 18 16:30:46 treeherder-stage app/release.6181: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.68.0.50/newrelic/hooks/database_dbapi2.py", line 22, in execute
Oct 18 16:30:46 treeherder-stage app/release.6181: *args, **kwargs)
Oct 18 16:30:46 treeherder-stage app/release.6181: File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 226, in execute
Oct 18 16:30:46 treeherder-stage app/release.6181: self.errorhandler(self, exc, value)
Oct 18 16:30:46 treeherder-stage app/release.6181: File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
Oct 18 16:30:46 treeherder-stage app/release.6181: raise errorvalue
Oct 18 16:30:46 treeherder-stage app/release.6181: django.db.utils.IntegrityError: (1062, "Duplicate entry '38ccf415-93e8-4911-939f-41b0bb44d043/0-77' for key 'text_log_summary_job_guid_1e839aa49a62d0a1_uniq'")
I could try to figure this out, but am tempted to just skip it. Does that seem reasonable James? I imagine you're thinking of just getting rid of this table entirely.
Flags: needinfo?(james)
Comment 19•8 years ago
|
||
Yes, I plan to delete the entire table in time; feel free to ignore it for now.
Flags: needinfo?(james)
Assignee | ||
Comment 20•8 years ago
|
||
Let's mark this as fixed.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•