I have verified that it should be straightforward to submit third-party performance data to perfherder. Proof of concept here (requires a currently uncommitted patch in bug 1183747): http://nbviewer.ipython.org/url/people.mozilla.org/~wlachance/test%20treeherder%20submit.ipynb (the key part is where we add the talos blob) There are some adjustments I'd like to make and I think we need to be careful about adding a flood of data (our list of performance signatures is already kinda big and we're currently not very smart about indexing it), but I think we'd be fine to add at least one or two performance runs on a single platform. Filing this bug to track this work.

This PR adds an artifact to perftest and populates it correctly for talostest. If this approach is good, I will do the same for robotest.py and then it is down to massaging the data for perfherder.

Comment on attachment 8668365 [details] https://github.com/mozilla/autophone/pull/39 updated the PR with all feedback and discussions.

Comment on attachment 8668365 [details] https://github.com/mozilla/autophone/pull/39 Looks good to me. Have you tested it yet?

you can see the "performance_data" bit in the artifact submission from the log: 2015-10-02 15:18:59,888|15568|MainThread|root|DEBUG|AutophoneTreeherder.submit_completed: tjc: [{"project": "mozilla-inbound", "job": {"submit_timestamp": 1443813358, "build_url": "http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/fennec-44.0a1.en-US.android-arm.apk", "machine_platform": {"platform": "android-4-1-armv7-api11", "os_name": "android", "architecture": "armv7"}, "result": "success", "end_timestamp": 1443813536, "build_platform": {"platform": "android-4-1-armv7-api11", "os_name": "android", "architecture": "armv7"}, "job_guid": "120cc91b-e1ad-4e79-9770-7d775b2af36a", "artifacts": [{"job_guid": "120cc91b-e1ad-4e79-9770-7d775b2af36a", "type": "json", "name": "text_log_summary", "blob": "{\"header\": {\"slave\": \"nexus-s\", \"revision\": \"5a9005eb43d15ee9bff1d107c542c7bb4fd3af45\"}, \"logname\": \"autophone-talos-tp4m-remote.ini-1-nexus-s-120cc91b-e1ad-4e79-9770-7d775b2af36a-autophone.log\", \"step_data\": {\"all_errors\": [], \"steps\": [{\"error_count\": 0, \"finished\": \"2015-10-02 15:18:56\", \"errors\": [], \"name\": \"step\", \"result\": \"success\", \"duration\": 178, \"order\": 0, \"started_linenumber\": 1, \"finished_linenumber\": 1}], \"errors_truncated\": false}, \"logurl\": \"https://autophone-dev.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/autophone-talos-tp4m-remote.ini-1-nexus-s-120cc91b-e1ad-4e79-9770-7d775b2af36a-autophone.log\"}"}, {"job_guid": "120cc91b-e1ad-4e79-9770-7d775b2af36a", "type": "json", "name": "Job Info", "blob": {"job_details": [{"content_type": "text", "value": "tp4m-remote.ini", "title": "Config"}, {"url": "http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/fennec-44.0a1.en-US.android-arm.apk", "content_type": "link", "value": "fennec-44.0a1.en-US.android-arm.apk", "title": "Build"}, {"content_type": "raw_html", "value": "1/0/0", "title": "Autophone Tp4m-tpn"}, {"url": "https://autophone-dev.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/autophone-talos-tp4m-remote.ini-1-nexus-s-120cc91b-e1ad-4e79-9770-7d775b2af36a-logcat.log", "content_type": "link", "value": "logcat", "title": "artifact uploaded"}, {"url": "https://autophone-dev.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/autophone-talos-tp4m-remote.ini-1-nexus-s-120cc91b-e1ad-4e79-9770-7d775b2af36a-autophone.log", "content_type": "link", "value": "Autophone Log", "title": "artifact uploaded"}]}}, {"job_guid": "120cc91b-e1ad-4e79-9770-7d775b2af36a", "type": "json", "name": "buildapi", "blob": {"buildername": "android-4-1-armv7-api11 mozilla-inbound opt autophone-talos"}}, {"job_guid": "120cc91b-e1ad-4e79-9770-7d775b2af36a", "type": "json", "name": "privatebuild", "blob": {"chunk": 1, "build_url": "http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/fennec-44.0a1.en-US.android-arm.apk", "config_file": "/home/jmaher/mozilla/autophone/configs/tp4m-remote.ini"}}, {"job_guid": "120cc91b-e1ad-4e79-9770-7d775b2af36a", "type": "json", "name": "performance_data", "blob": "{\"suites\": [{\"subtests\": [{\"name\": \"m.wikipedia.com\", \"value\": 578.0}, {\"name\": \"m.bbc.co.uk\", \"value\": 492.0}, {\"name\": \"m.yahoo.co.jp\", \"value\": 1636.0}, {\"name\": \"m.espn.com\", \"value\": 856.0}, {\"name\": \"amazon.com\", \"value\": 1882.0}, {\"name\": \"m.yandex.ru\", \"value\": 1099.5}, {\"name\": \"m.accuweather.com\", \"value\": 378.0}], \"name\": \"remote-tp4m\", \"value\": 846.6830685714009}]}"}], "machine": "nexus-s", "state": "completed", "product_name": "fennec", "option_collection": {"opt": true}, "who": "", "group_symbol": "A", "reason": "", "tier": 3, "group_name": "Autophone", "desc": "", "start_timestamp": 1443813358, "name": "Autophone Tp4m", "log_references": [{"url": "https://autophone-dev.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/1443810517/autophone-talos-tp4m-remote.ini-1-nexus-s-120cc91b-e1ad-4e79-9770-7d775b2af36a-autophone.log", "parse_status": "parsed", "name": "autophone-talos-tp4m-remote.ini-1-nexus-s-120cc91b-e1ad-4e79-9770-7d775b2af36a-autophone.log"}], "job_symbol": "tpn"}, "revision_hash": "5a9005eb43d15ee9bff1d107c542c7bb4fd3af45", "coalesced": []}]

okie dokie.

landed: https://github.com/mozilla/autophone/commit/af57e65369cc956f74cabf48a58c75f43c715a1a we still need to see this on a live server, testing treeherder locally is sort of like gambling. Next week this should be awesome.

(In reply to Joel Maher (:jmaher) from comment #7) > landed: > https://github.com/mozilla/autophone/commit/ > af57e65369cc956f74cabf48a58c75f43c715a1a > > we still need to see this on a live server, testing treeherder locally is > sort of like gambling. Next week this should be awesome. This landed as part of the bug 1209796 deployment. It hasn't been enabled yet. We'll have to update the production-manifest.ini when you are ready.

https://github.com/mozilla/autophone/commit/67ec852b7dca1e605096b58772780f4c06590b38 I had to revert this since the autophonetreeherder changes broke the non Perf tests with the test: if t.perfherder_artifact We should have checked for the attribute and if it was non-None. I totally missed this in the review. Sorry.

updated pull request with try/except around block to catch attributeerror if the attribute doesn't exist.

Comment on attachment 8671550 [details] https://github.com/mozilla/autophone/pull/40 r+. Talk to me before you push this please.

landed: https://github.com/mozilla/autophone/commit/3a896456a8128f755de1a73aaafd15f3c5598d6f