"Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test mochitest-browser-chrome-7" is missing from allthethings.json I definitely see the job on TH [1] I don't see the job in the production allthethings.json [2] but I see it in the one I generated locally [3] The size of the files are different [4] This is important to fix as it affects the normal behaviour of tools we have built. [1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=rev5%20macosx%20yosemite%2010.10%20mozilla-inbound%20debug%20test%20mochitest-browser-chrome-7 [2] armenzg@armenzg-thinkpad:~/repos/dump/community$ wget https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json armenzg@armenzg-thinkpad:~/repos/dump/community$ grep "Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test mochitest-browser-chrome-7" allthethings.json [3] armenzg@armenzg-thinkpad:~/repos/dump/community$ grep "Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test mochitest-browser-chrome-7" ~/.mozilla/releng/repos/buildbot-configs/allthethings.json "Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test mochitest-browser-chrome-7": { "Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test mochitest-browser-chrome-7", "Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test mochitest-browser-chrome-7", [4] armenzg@armenzg-thinkpad:~/repos/dump/community$ ls -l allthethings.json -rw-rw-r-- 1 armenzg armenzg 11333972 Oct 1 09:55 allthethings.json armenzg@armenzg-thinkpad:~/repos/dump/community$ ls -l ~/.mozilla/releng/repos/buildbot-configs/allthethings.json -rw------- 1 armenzg armenzg 11811616 Oct 1 09:37 /home/armenzg/.mozilla/releng/repos/buildbot-configs/allthethings.json

<catlee-mtg> armenzg: hg was hung for 16h Also see bug 1131787 for making it a production artifact (which is not now).

Callek, if you end up working on setting up an alert please let us know which bug.

I'm finding discrepancies between the file officially generated and the one I'm generating locally. catlee: could you please paste in here the code that generates this file so I can compare? and under which recurrence? Or is hg hang again? Or can I please have access to the machine? I think a simple diff is not a good tool to look for discrepancies as I get the feeling that the discrepancies are coming from the SETA schedulers.

we're using this code: http://hg.mozilla.org/build/braindump/file/b807e156e984/buildbot-related/dump_allthethings.sh and it's running on cruncher. do you still have access to that machine?

Is this the script that updates it? https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Allthethings If so, it would like the file gets generated based on tip. I don't know the host name anymore. I've filed a service now ticket as I can't connect to the VPN atm

I've managed to ssh as myself. How do I get to the buildduty user?

Armen, what host are you on? sudo su - buildduty usually works, but not sure if you have the permissions we can run it for if you don't have permissions

Thanks kmoir. I've documented the steps: https://wiki.mozilla.org/ReleaseEngineering/How_To/allthethings.json#Debug_in_production I'm modifying my script and request that we switch to it when I get it to par to catlee's script.

At the moment, I don't know why catlee's setup would generate a different allthethings.json [1] I will look more into it next week. I'm setting my account up so it can generate the file using generate_allthethings_json.sh (community friendly). This is what I have so far under my account: Installed Python 2.7.10: mkdir ~/dist wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz tar -zxvf Python-2.7.10.tgz cd Python-2.7.10 ./configure --prefix=/home/armenzg/python_install make && make install Add new python to my PATH Added $HOME/python_install/bin to my PATH in .bash_profile Installed virtualenv: wget https://pypi.python.org/packages/source/v/virtualenv/virtualenv-13.1.2.tar.gz tar xvfz virtualenv-13.1.2.tar.gz cd virtualenv-13.1.2/ python setup.py install [1] By doing all this, when virtualenv is called it in my script it won't use Python 2.6.6. My virtualenv generation script uses -p `which python` so /usr/bin/python is not used Run script: hg clone https://hg.mozilla.org/build/braindump ~/braindump/community/generate_allthethings_json.sh [1] (allthethings)[catlee@cruncher.srv.releng.scl3 allthethings]$ pip freeze Jinja2==2.5.5 MozillaPulse==0.80 MySQL-python==1.2.3 SQLAlchemy==0.6.4 Twisted==10.1.0 amqp==1.4.4 amqplib==0.6.1 anyjson==0.3 argparse==1.1 buildbot==0.8.2-hg-d079c533f995-production-0.8 carrot==0.10.7 distribute==0.6.24 kombu==3.0.14 pyOpenSSL==0.10 pyasn1==0.0.11a pycrypto==2.3 pytz==2011d wsgiref==0.1.2 zope.interface==3.6.1 (allthethings)[catlee@cruncher.srv.releng.scl3 allthethings]$ hg -R tools id e6635fd205b3 tip (allthethings)[catlee@cruncher.srv.releng.scl3 allthethings]$ hg -R buildbot-configs id 573ffaae579f (production) (allthethings)[catlee@cruncher.srv.releng.scl3 allthethings]$ hg -R buildbotcustom id b65894463876 (production-0.8) (allthethings)[catlee@cruncher.srv.releng.scl3 allthethings]$ python --version Python 2.7.3 [2] [armenzg@cruncher.srv.releng.scl3 virtualenv-13.1.2]$ python setup.py install running install running build running build_py running build_scripts copying and adjusting scripts/virtualenv-2.7 -> build/scripts-2.7 running install_lib copying build/lib/virtualenv.py -> /home/armenzg/python_install/lib/python2.7/site-packages creating /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support copying build/lib/virtualenv_support/setuptools-18.2-py2.py3-none-any.whl -> /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support copying build/lib/virtualenv_support/wheel-0.24.0-py2.py3-none-any.whl -> /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support copying build/lib/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl -> /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support copying build/lib/virtualenv_support/__init__.py -> /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support copying build/lib/virtualenv_support/argparse-1.3.0-py2.py3-none-any.whl -> /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support byte-compiling /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv.py to virtualenv.pyc byte-compiling /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv_support/__init__.py to __init__.pyc running install_scripts copying build/scripts-2.7/virtualenv -> /home/armenzg/python_install/bin copying build/scripts-2.7/virtualenv-2.7 -> /home/armenzg/python_install/bin changing mode of /home/armenzg/python_install/bin/virtualenv to 755 changing mode of /home/armenzg/python_install/bin/virtualenv-2.7 to 755 running install_egg_info Writing /home/armenzg/python_install/lib/python2.7/site-packages/virtualenv-13.1.2-py2.7.egg-info

I don't exactly know what the root cause could have been of the differences. Here's a diff of the venv's. I started generating this new file: https://secure.pub.build.mozilla.org/builddata/reports/allthethings.new.json I have a script that will generate it on the same schedule as catlee's set up so I can be told of any differences. This is my crontab: */15 * * * * $HOME/braindump/community/generate_allthethings_json.sh | mail -E -s '[allthethings] all the things' armenzg@mozilla.com The logic to determine if changes have happened is in here: https://hg.mozilla.org/build/braindump/file/a4bcdd653a8e/community/generate_allthethings_json.sh#l37 At this point in time I don't believe there should have been enough differences to be realistic that there were issues in here (we'll see if my change detection script reveals anything). I believe one issue was caching issues on pulse_actions: https://github.com/mozilla/mozilla_ci_tools/issues/365 Nevertheless, my detection script should have not been triggered since it is unrelated to pulse_actions.

catlee: I believe the original issue was due to hg being hung and I hope I won't have anymore discrepancies. Nevertheless, would you like to switch over to using generate_allthethings_json.sh as the main entry point? (since it is used by community and the set up of the environment is nailed by setup_buildbot_environment.sh) I can set it up under the buildduty account. We could also ask on bug 1212002 for someone to drip hgtool.py into braindump and use that for better handling. What do you say?

FTR I undid my custom set up from https://bugzilla.mozilla.org/show_bug.cgi?id=1210390#c9

Sadly I discovered that allthethings.json is out of date not at most 15 minutes (the cronjob) but as long as the rsync from cruncher happens (every hour around 10 minutes past). I think we should have a pulse listener that schedules a TC task which generates the file and makes it publicly available. That is, if we want to have a proper automatic backfilling system (which would not be able to determine if a builder if valid or not). Hrmm perhaps the system should assume the builder is valid to begin with an try to schedule it.

Sure, please feel free to use the same code path as the community scripts use.

Made the changes in bug 1219390.