I have seen this before so this is nothing new. We *probably* avoid this in production since IIRC we flash the device for each run. In any case, I make a ts testing file like:: python remotePerfConfigurator.py -v -e org.mozilla.fennec --develop --activeTests ts --output ts.yml --results_url file://${HOME}/ts.txt --noChrome --remoteDevice 10.251.28.71 The resulting file:: # Sample Talos configuration file # The title of the report title: qm-pxp01 test_name_extension: _nochrome #*** output options **** #uncomment to turn on dump to csv option #csv_dir: 'output' #comment out next line to disable send to graph server results_url: file:///home/jhammel/ts.txt develop: True fennecIDs: # data filters # filters: [median] # browser info process: fennec browser_path: org.mozilla.fennec browser_log: browser_output.txt # arguments to pass to browser extra_args: '' #how long the browser takes to open/close browser_wait: 5 test_timeout: 1200 bcontroller_config: ${talos}/bcontroller.yml branch: mobile remote: True deviceip: 10.251.28.71 deviceport: 20701 webserver: 10.251.27.101:15707 deviceroot: /mnt/sdcard/tests buildid: 2012030603 init_url: http://10.251.27.101:15707/getInfo.html # Preferences to set in the test (use "preferences : {}" for no prefs) preferences: extensions.enabledScopes: 5 extensions.autoDisableScopes: 10 browser.EULA.override : true security.fileuri.strict_origin_policy : false browser.shell.checkDefaultBrowser : false browser.warnOnQuit : false browser.link.open_newwindow : 2 dom.allow_scripts_to_close_windows : true dom.disable_open_during_load: false dom.max_script_run_time : 0 hangmonitor.timeout : 0 browser.dom.window.dump.enabled: true dom.disable_window_flip : true dom.disable_window_move_resize : true security.enable_java : false extensions.checkCompatibility : false extensions.update.notifyUser: false talos.logfile: /mnt/sdcard/tests/browser_output.txt browser.cache.disk.smart_size.enabled: false browser.cache.disk.smart_size.firstrun: false network.manage-offline-status: false extensions: - ${talos}/pageloader extensions: - ${talos}/pageloader #any directories whose contents need to be installed in the browser before running the tests # this assumes that the directories themselves already exist in the firefox path dirs: chrome : ${talos}/page_load_test/chrome components : ${talos}/page_load_test/components # Environment variables to set during test (use env: {} for none) env : NO_EM_RESTART : 1 MOZ_CRASHREPORTER_NO_REPORT : 1 MOZ_CRASHREPORTER_SHUTDOWN : 1 # Tests to run # url : (REQUIRED) url to load into the given firefox browser # url_mod : (OPTIONAL) a bit of code to be evaled and added to the given url during each cycle of the test # resolution : (REQUIRED) how long (in seconds) to pause between counter sampling # cycles : (REQUIRED) how many times to run the test # timeout : (OPTIONAL) how many seconds the test can run before we consider it failed and quit (default 8 hours) # pagetimeout : (OPTIONAL) how many seconds each page is allowed to take before considered to be frozen (default 8 hours) # Must be used in conjuction with the pageloader with the -tpnoisy option # counters : (REQUIRED) types of system activity to monitor during test run, can be empty # For possible values of counters argument on Windows, see # http://technet2.microsoft.com/WindowsServer/en/Library/86b5d116-6fb3-427b-af8c-9077162125fe1033.mspx?mfr=true # Possible values on Mac: # counters : ['Private Bytes', 'RSS'] # Possible values on Linux: # counters : ['Private Bytes', 'RSS', 'XRes'] # Standard windows values: # counters : ['Working Set', 'Private Bytes', '% Processor Time'] # to set up a new test it must have the correct configuration options and drop information in a standard format # the format is seen in the regular expressions in ttest.py # to see how the data passed from the browser is processed see send_to_graph and send_to_csv in run_tests.py basetest: url : url_mod : resolution : 1 cycles : 1 timeout : 3600 win_counters : [] w7_counters : [] linux_counters : [] mac_counters : [] remote_counters : [] shutdown : False responsiveness : False profile_path : ${talos}/mobile_profile # pageloader options tpmanifest: tpchrome: True tpmozafterpaint: False tpnoisy: True tpformat: tinderbox tpcycles: 3 tppagecycles: 1 rss: False tprender: False tpdelay: 1000 tests : - name: ts tpchrome: False url : http://10.251.27.101:15707/startup_test/startup_test.html?begin= url_mod : str(int(time.time()*1000)) cycles : 10 timeout: 300 shutdown : True Then I run it like:: python run_tests.py -n -d ts.yml However, I get output like tsvg, which was the previous test I ran:: <snip/> NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-001.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml) NOISE: RSS: Main: 83058688 NOISE: NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml) NOISE: RSS: Main: 81244160 NOISE: NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml) NOISE: RSS: Main: 81510400 NOISE: NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml) NOISE: RSS: Main: 79773696 NOISE: NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml) NOISE: RSS: Main: 80232448 NOISE: NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml) NOISE: RSS: Main: 77053952 NOISE: NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml (next: http://10.251.27.101:15707/page_load_test/svg/gearflowers.svg) NOISE: RSS: Main: 78024704 NOISE: NOISE: RSS: Main: 78012416 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/gearflowers.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale.svg) NOISE: RSS: Main: 78225408 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale-opacity.svg) NOISE: RSS: Main: 80527360 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale-opacity.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate.svg) NOISE: RSS: Main: 78508032 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate-opacity.svg) NOISE: RSS: Main: 78635008 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate-opacity.svg (next: http://10.251.27.101:15707/page_load_test/svg/hixie-001.xml) NOISE: RSS: Main: 78962688 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-001.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml) NOISE: RSS: Main: 79032320 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml) NOISE: RSS: Main: 73363456 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml) NOISE: RSS: Main: 79794176 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml) NOISE: RSS: Main: 76701696 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml) NOISE: RSS: Main: 77381632 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml) NOISE: RSS: Main: 77598720 NOISE: NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml (next: http://10.251.27.101:15707/page_load_test/svg/gearflowers.svg) NOISE: RSS: Main: 79126528 NOISE: NOISE: __start_tp_report NOISE: _x_x_mozilla_page_load,3444.9545454545455,NaN,NaN NOISE: _x_x_mozilla_page_load_details,avgmedian|3444.9545454545455|average|3444.95|minimum|NaN|maximum|NaN|stddev|NaN NOISE: |i|pagename|median|mean|min|max|runs| NOISE: |0;gearflowers.svg;664;664;541;853;787;853;541 NOISE: |1;composite-scale.svg;206;206;201;283;211;283;201 NOISE: |2;composite-scale-opacity.svg;49.5;49.5;45;58;58;54;45 NOISE: |3;composite-scale-rotate.svg;40;40;38;49;49;38;42 NOISE: |4;composite-scale-rotate-opacity.svg;50;50;50;81;50;81;50 NOISE: |5;hixie-001.xml;15137.5;15137.5;15122;15182;15182;15153;15122 NOISE: |6;hixie-002.xml;15123.5;15123.5;15113;15166;15134;15113;15166 NOISE: |7;hixie-003.xml;5061.5;5061.5;5055;5083;5055;5083;5068 NOISE: |8;hixie-004.xml;5159;5159;5158;5167;5160;5158;5167 NOISE: |9;hixie-005.xml;5124.5;5124.5;5124;5145;5125;5145;5124 NOISE: |10;hixie-006.xml;5109.5;5109.5;5097;5139;5097;5139;5122 NOISE: |11;hixie-007.xml;1307;1307;1281;1368;1333;1368;1281 NOISE: __end_tp_report NOISE: __start_cc_report NOISE: _x_x_mozilla_cycle_collect,1493 NOISE: __end_cc_report NOISE: __startTimestamp1331072013846__endTimestamp NOISE: NOISE: __metrics Screen width/height:0/0 NOISE: colorDepth:24 NOISE: Browser inner width/height: 980/800 NOISE: __metrics__start_report2643__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072590349 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072588244" line: 0}] NOISE: __startTimestamp1331072591155__endTimestamp NOISE: __start_report2652__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072636105 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072633893" line: 0}] NOISE: __startTimestamp1331072636774__endTimestamp NOISE: __start_report3304__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072682424 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072679556" line: 0}] NOISE: __startTimestamp1331072682943__endTimestamp NOISE: __start_report2755__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072727961 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072725674" line: 0}] NOISE: __startTimestamp1331072728626__endTimestamp NOISE: __start_report2902__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072781122 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072778674" line: 0}] NOISE: __startTimestamp1331072781701__endTimestamp NOISE: __start_report2749__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072835213 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072833082" line: 0}] NOISE: __startTimestamp1331072835884__endTimestamp NOISE: __start_report2951__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072880816 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072878647" line: 0}] NOISE: __startTimestamp1331072881640__endTimestamp NOISE: __start_report2954__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072927455 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072925258" line: 0}] NOISE: __startTimestamp1331072928419__endTimestamp NOISE: __start_report3094__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331072974147 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072971733" line: 0}] NOISE: __startTimestamp1331072974860__endTimestamp NOISE: __start_report2719__end_report NOISE: NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'. NOISE: zerdatime 1331073019954 - browser chrome startup finished. NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated. Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331073017687" line: 0}] NOISE: __startTimestamp1331073020499__endTimestamp getting files in '/mnt/sdcard/tests/profile/minidumps/' DEBUG: Received test results: 2215 2215 2215 2215 2215 2215 2215 2215 2215 2215 Completed test ts: Stopped Tue, 06 Mar 2012 14:33:24 RETURN: cycle time: 00:08:44<br> qm-pxp01: Stopped Tue, 06 Mar 2012 14:33:24 Sending results: Started Tue, 06 Mar 2012 14:33:24 DEBUG: Working with test: ts DEBUG: Sending results: 2215 2215 2215 2215 2215 2215 2215 2215 2215 2215 Generating results file: ts: Started Tue, 06 Mar 2012 14:33:24 Generating results file: ts: Stopped Tue, 06 Mar 2012 14:33:24 Generating results file: ts_shutdown: Started Tue, 06 Mar 2012 14:33:24 Generating results file: ts_shutdown: Stopped Tue, 06 Mar 2012 14:33:24 Transmitting test: ts: Started Tue, 06 Mar 2012 14:33:24 Transmitting test: ts: Stopped Tue, 06 Mar 2012 14:33:24 Transmitting test: ts_shutdown: Started Tue, 06 Mar 2012 14:33:24 Transmitting test: ts_shutdown: Stopped Tue, 06 Mar 2012 14:33:24 RETURN:<br> RETURN:<p style="font-size:smaller;">Details:<br>|</p> Completed sending results: Stopped Tue, 06 Mar 2012 14:33:24 There is some test_startup.html in there but there is also some tsvg pages like http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml . I don't know what's going on here, but needless to say it worries me.

Running it twice seems to "clear" whatever is happening that has leftover tsvg stuff

Beh, I lied, it does not

Neither does resetting the device

looking in /mnt/sdcard/tests svg.manifest does linger after the tsvg run. I haven't tracked how it is being picked up by talos yet. Maybe the thing to do is nuke it

clearing out /mnt/sdcard/{svg.manifest,browser.log} seems to fix this problem. Maybe we should clear out this directory before testing as well as afterwards?

actually you only need to clear out browser.log and all of this should live in the /mnt/sdcard/tests folder. Anyway, this should take place in run_tests.py or something like that.

we can probably fix this bug now without much trouble.

moving the remaining android talos tests to autophone this quarter, autophone is more robust in device management and retrying, most likely we will not see this issue there.