eg: Oct 19 15:14:29 treeherder app/web.1: [2015-10-19 07:14:28,790] ERROR [treeherder.webapp.api.exceptions:26] (1054, "Unknown column 'performance_signature.repository_id' in 'field list'") Oct 19 15:14:29 treeherder app/web.1: [2015-10-19 07:14:28,791] ERROR [treeherder.webapp.api.exceptions:27] Traceback (most recent call last): Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch Oct 19 15:14:29 treeherder app/web.1: response = handler(request, *args, **kwargs) Oct 19 15:14:29 treeherder app/web.1: File "/app/treeherder/webapp/api/artifact.py", line 65, in create Oct 19 15:14:29 treeherder app/web.1: artifacts_model.load_job_artifacts(artifacts, job_id_lookup) Oct 19 15:14:29 treeherder app/web.1: File "/app/treeherder/model/derived/artifacts.py", line 183, in load_job_artifacts Oct 19 15:14:29 treeherder app/web.1: performance_artifact_job_id_list, performance_artifact_list) Oct 19 15:14:29 treeherder app/web.1: File "/app/treeherder/model/derived/artifacts.py", line 123, in store_performance_artifact Oct 19 15:14:29 treeherder app/web.1: load_talos_artifacts(self.project, ref_data, job_data, perf_data) Oct 19 15:14:29 treeherder app/web.1: File "/app/treeherder/etl/perf.py", line 258, in load_talos_artifacts Oct 19 15:14:29 treeherder app/web.1: 'last_updated': push_timestamp Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method Oct 19 15:14:29 treeherder app/web.1: return getattr(self.get_queryset(), name)(*args, **kwargs) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 405, in get_or_create Oct 19 15:14:29 treeherder app/web.1: return self.get(**lookup), False Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get Oct 19 15:14:29 treeherder app/web.1: num = len(clone) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__ Oct 19 15:14:29 treeherder app/web.1: self._fetch_all() Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all Oct 19 15:14:29 treeherder app/web.1: self._result_cache = list(self.iterator()) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator Oct 19 15:14:29 treeherder app/web.1: results = compiler.execute_sql() Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql Oct 19 15:14:29 treeherder app/web.1: cursor.execute(sql, params) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute Oct 19 15:14:29 treeherder app/web.1: return self.cursor.execute(sql, params) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__ Oct 19 15:14:29 treeherder app/web.1: six.reraise(dj_exc_type, dj_exc_value, traceback) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute Oct 19 15:14:29 treeherder app/web.1: return self.cursor.execute(sql, params) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute Oct 19 15:14:29 treeherder app/web.1: return self.cursor.execute(query, args) Oct 19 15:14:29 taskcluster-queue app/web.10: Mon, 19 Oct 2015 14:14:28 GMT taskcluster-client Success calling: authenticateHawk, (0 retries) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/hooks/database_dbapi2.py", line 22, in execute Oct 19 15:14:29 treeherder app/web.1: *args, **kwargs) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute Oct 19 15:14:29 treeherder app/web.1: self.errorhandler(self, exc, value) Oct 19 15:14:29 treeherder app/web.1: File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler Oct 19 15:14:29 treeherder app/web.1: raise errorclass, errorvalue Oct 19 15:14:29 treeherder app/web.1: OperationalError: (1054, "Unknown column 'performance_signature.repository_id' in 'field list'") https://papertrailapp.com/groups/853883/events?q=program%3Aapp%2Fweb.1&r=592767746937499660-592767773323866114 https://rpm.newrelic.com/accounts/677903/applications/7385291/traced_errors

Looking at the database, it appears that the migration didn't happen. Specifically this one: https://github.com/mozilla/treeherder/blob/master/treeherder/perf/migrations/0002_auto_20151014_0922.py mysql> describe performance_signature; +----------------------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------------------+-------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | signature_hash | varchar(40) | NO | UNI | NULL | | | suite | varchar(80) | NO | | NULL | | | test | varchar(80) | NO | | NULL | | | extra_properties | longtext | NO | | NULL | | | framework_id | int(11) | NO | MUL | NULL | | | option_collection_id | int(11) | NO | MUL | NULL | | | platform_id | int(11) | NO | MUL | NULL | | +----------------------+-------------+------+-----+---------+----------------+ 8 rows in set (0.07 sec) There should be repository_id and last_updated fields in that table after the migration is applied.

manage.py migrate isn't run automatically on Heroku on the moment, that's bug 1201455. However running manually fails with: [~/src/treeherder]$ heroku run ./manage.py migrate Running ./manage.py migrate on treeherder-heroku... up, run.3116 Operations to perform: Synchronize unmigrated apps: django_browserid, log_parser, staticfiles, autoclassify, corsheaders, workers, rest_framework_swagger, messages, etl, rest_framework_extensions, embed, rest_framework, webapp, hawkrest Apply all migrations: sessions, perf, admin, auth, sites, contenttypes, model, credentials Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying credentials.0002_auto_20151014_0900... OK Applying model.0002_auto_20151014_0900...Traceback (most recent call last): File "./manage.py", line 11, in <module> execute_from_command_line(sys.argv) File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line utility.execute() File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv self.execute(*args, **cmd_options) File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute output = self.handle(*args, **options) File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 222, in handle executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/executor.py", line 110, in migrate self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/executor.py", line 148, in apply_migration state = migration.apply(state, schema_editor) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/migration.py", line 115, in apply operation.database_forwards(self.app_label, schema_editor, old_state, project_state) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 201, in database_forwards schema_editor.alter_field(from_model, from_field, to_field) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 484, in alter_field old_db_params, new_db_params, strict) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 493, in _alter_field fk_names = self._constraint_names(model, [old_field.column], foreign_key=True) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 913, in _constraint_names constraints = self.connection.introspection.get_constraints(cursor, model._meta.db_table) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/introspection.py", line 190, in get_constraints cursor.execute("SHOW INDEX FROM %s" % self.connection.ops.quote_name(table_name)) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute return self.cursor.execute(sql) File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute return self.cursor.execute(query, args) File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue django.db.utils.ProgrammingError: (1146, "Table 'treeherder.failure_match' doesn't exist") Looks like not all of the manual changes that people have been making to stage/prod were also made to the Heroku RDS instance - guessing this needs the table adding that landed in bug 1182464.

This is a treeherder/heroku problem, not a perfherder one.

(In reply to William Lachance (:wlach) from comment #3) > This is a treeherder/heroku problem, not a perfherder one. This is a combination of two problems: * Manual changes having not been made against the Heroku RDS instance previously (for which I've just filed bug 1217631), which has broken migrate. * Migrate having not been run manually against Heroku since it's non-obvious that it doesn't yet happen automatically. I'll try to get the migrate to happen as part of deploy as soon as possible (bug 1201455). However for things for which we can't use migrate and so have to update stage/prod by hand too, it would be really useful if people could remember to update Heroku at the same time, otherwise it takes twice as long for me to figure out what's broken and which changes need to be applied (particularly when they aren't documented in the bugs :-().

[~/src/treeherder]$ heroku run ./manage.py migrate Running ./manage.py migrate on treeherder-heroku... up, run.6381 Operations to perform: Synchronize unmigrated apps: django_browserid, log_parser, rest_framework_extensions, autoclassify, rest_framework, workers, rest_framework_swagger, hawkrest, messages, embed, staticfiles, etl, webapp, corsheaders Apply all migrations: credentials, perf, model, auth, sites, admin, sessions, contenttypes Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying model.0002_auto_20151014_0900... OK Applying perf.0002_auto_20151014_0922... OK The following content types are stale and need to be deleted: model | device Any objects related to these content types by a foreign key will also be deleted. Are you sure you want to delete these content types? If you're unsure, answer 'no'. Type 'yes' to continue, or 'no' to cancel: yes [~/src/treeherder]$ heroku run ./manage.py migrate Running ./manage.py migrate on treeherder-heroku... up, run.7866 Operations to perform: Synchronize unmigrated apps: rest_framework_swagger, embed, log_parser, django_browserid, messages, rest_framework_extensions, etl, rest_framework, hawkrest, staticfiles, workers, autoclassify, corsheaders, webapp Apply all migrations: auth, sites, model, contenttypes, sessions, admin, perf, credentials Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: No migrations to apply.