Closed
Bug 827497
Opened 12 years ago
Closed 12 years ago
Modifications to the W64 task sequence
Categories
(Infrastructure & Operations :: RelOps: General, task)
Infrastructure & Operations
RelOps: General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dustin, Assigned: dustin)
References
Details
Attachments
(1 file)
(deleted),
image/png
|
Details |
* install the 8.0 SDK along with a patch for bug 774910
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=29327649 suggests the following be done manually; they should be automatic:
* set the local administrator password during the task sequence (without including it in cleartext!)
* make the server-management app not appear on startup
* fix the drive mapping (second drive at e:, CD-ROM elsewhere)
* set up autologin for the cltbld user; if possible, don't include the user's cleartext password here
Comment 1•12 years ago
|
||
COMPLETED - install the 8.0 SDK along with a patch for bug 774910
COMPLETED - make the server-management app not appear on startup
COMPLETED - fix the drive mapping (second drive at e:, CD-ROM elsewhere), this is dependent upon each hardware configuration
COMPLETED - set up autologin for the cltbld user; if possible, don't include the user's cleartext password here
Assignee | ||
Comment 2•12 years ago
|
||
Great - I'll try reimaging the five machines we have waiting for this, and hand them to releng to have a look at.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 3•12 years ago
|
||
Bad news :(
This is failing to run the "Upgrade Buildbot" step. Running the script manually permission denied errors.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 4•12 years ago
|
||
So from what I see, the things to fix are:
1. Convert the batch file to an MDT application for better control
2. Don't try to run as an administrator (not necessary)
3. Move the BinScope MSI install into a separate TS step
4. Change the 'md' at the end to use E:
5. In a new step, copy http://hg.mozilla.org/build/puppet/file/tip/modules/buildslave/files/runslave.py to C:\ (make a local copy of the script in the TS, rather than fetching it from that URL).
Hopefully that should fix the problem.
Comment 5•12 years ago
|
||
Moved the BuildBot Upgrade into a MDT Application.
Removed all wget calls from batch and instead we copy down an updated source at start then run the commands locally. This removes the external dependencies for the imaging process.
MD is still set to use D:, because we don't shuffle the drives around until after MDT completes (After-action script)
Runslave.py is copied down as part of the MDT application.
Assignee | ||
Comment 6•12 years ago
|
||
I'm running this (from the production deployment share) on w64-ix-slave21 now.
Assignee | ||
Comment 7•12 years ago
|
||
No luck: bug 820992 comment 6:
> http://dev-master01.build.scl1.mozilla.com:8052/builders/WINNT%205.
> 2%20mozilla-central%20xulrunner/builds/1/steps/clean_old_builds/logs/stdio
>
> Traceback (most recent call last):
> File "tools/buildfarm/maintenance/purge_builds.py", line 31, in <module>
> import win32file
> ImportError: No module named win32file
>
> while running
>
> 'python' 'tools/buildfarm/maintenance/purge_builds.py' '-s' '12' '-n' 'info'
> '-n' 'rel-*' '..' '/mock/users/cltbld/home/cltbld/build'
>
> with a PATH of
>
> PATH=C:\mozilla-build\nsis-2.46u;C:\mozilla-build\python27;C:\mozilla-
> build\buildbotve\scripts;C:\mozilla-build\msys\local\bin;c:\mozilla-
> build\wget;c:\mozilla-build\7zip;c:\mozilla-build\blat261\full;c:\mozilla-
> build\python;c:\mozilla-build\svn-win32-1.6.3\bin;c:\mozilla-build\upx203w;c:
> \mozilla-build\emacs-22.3\bin;c:\mozilla-build\info-zip;c:\mozilla-
> build\nsis-2.22;c:\mozilla-build\nsis-2.33u;c:\mozilla-build\nsis-2.46u;c:
> \mozilla-build\wix-351728;c:\mozilla-build\hg;c:\mozilla-
> build\python\Scripts;c:\mozilla-build\kdiff3;c:\mozilla-build\yasm;.;C:
> \mozilla-build\msys\local\bin;C:\mozilla-build\msys\mingw\bin;C:\mozilla-
> build\msys\bin;c:\Program Files (x86)\Microsoft Visual Studio
> 9.0\VC\BIN\amd64;c:\Windows\Microsoft.NET\Framework64\v3.5;c:
> \Windows\Microsoft.NET\Framework64\v3.5\Microsoft .NET Framework 3.5
> (Pre-Release
> Version);c:\Windows\Microsoft.NET\Framework64\v2.0.50727;c:\Program Files
> (x86)\Microsoft Visual Studio
> 9.0\VC\VCPackages;c:\Tools\msvs9\Common7\IDE;c:\Tools\msvs9\Common7\Tools;c:
> \Tools\msvs9\Common7\Tools\bin;c:\Tools\sdks\v7.0\bin\x64;c:\Tools\sdks\v7.
> 0\bin\win64\x64;c:\Tools\sdks\v7.0\bin;c:\windows\System32;c:\windows;c:
> \windows\System32\Wbem;c:\mozilla-build\moztools-x64\bin;c:\mozilla-
> build\vim\vim72
> Do you know where win32file is expected to be installed? As in, which
> Python? Py27 is in that path before buildbotve, so it's being run instead
> of the buildbotve python.
>
> I don't think we have any steps to install win32file. I don't see it in
> https://wiki.mozilla.org/ReferencePlatforms/Win64
> I'm able to import win32file from c:\mozilla-build\python27\python.exe, and
> I see a win32file.pyd in c:\mozilla-build\python27\Lib\site-packages on
> w64-ix-slave22.
>
> I'm unfamiliar with what it is or why we need it; I'll ask around.
> Evidently it's from pywin32, which is on that refplatform page:
> https://wiki.mozilla.org/ReferencePlatforms/Win64#Python_2.7.3
That's new stuff from bug 602908, I think. The MSI install was in buildbot_update.bat, so that got done, but the second part was not.
This means adding the equivalent of
c:\mozilla-build\msys\bin\bash --login -i
/c/mozilla-build/wget/wget.exe http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/python/install_pywin32_python27.sh
./install_pywin32_python27.sh
to the install process. The content of that shell script is
----
MOZILLABUILD=/c/mozilla-build
if [ -d /d/mozilla-build ]; then
MOZILLABUILD=/d/mozilla-build
fi
rm -rf pywin32-temp
mkdir pywin32-temp
cd pywin32-temp
${MOZILLABUILD}/wget/wget http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/python/pywin32-218.win32-py2.7.exe
${MOZILLABUILD}/info-zip/unzip pywin32-218.win32-py2.7.exe
cp -r PLATLIB/* ${MOZILLABUILD}/python27/lib/site-packages
cp -r SCRIPTS/* ${MOZILLABUILD}/python27/lib/site-packages
pushd ${MOZILLABUILD}/python27/lib/site-packages
${MOZILLABUILD}/python27/python pywin32_postinstall.py -quiet -install
popd
cd ..
rm -rf pywin32-temp
----
This is an abomination, I know..
Basically that's getting pywin32-218.win32-py2.7.exe (which we'll get in \\releng\data\apps\tmp\bug827497 for you, Chris), unzipping it, and copying the contents of PLATLIB and SCRIPTS into c:\mozillabuild\python27\lib\site-packages. Then it changes directory to c:\mozillabuild\python27\lib\site-packages and runs pywin32_postinstall.py with c:\mozilla-build\python27\python.exe. I'm assuming the directory-change is critical there.
So: Chris, can you replicate that? You can expense the anti-nausea pills.
Aki/Coop: are there other surprises from around the time this was implemented or later, that we can head off before we re-image and hand back to you again?
Comment 8•12 years ago
|
||
Updated the build to account for this app:
msiexec /i %mozillabuild%\python-2.7.3.msi /qb /norestart /log c:\logs\python27_install.log TARGETDIR=c:\mozilla-build\python27 ALLUSERS=1
SET python27=%mozillabuild%\python27
%python27%\python.exe %python27%\lib\site-packages\pywin32_postinstall.py -quiet -install
Assignee | ||
Comment 9•12 years ago
|
||
Indeed, I can now import 'win32file' using c:\mozilla-build\python27\python.exe.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 10•12 years ago
|
||
So, on one that ran a job already (w64-ix-slave11), this has a firewall warning.
Coop, let's talk on Tuesday about this Python-2.7.3 project. I'm not sure where the project became de-synchronized with the MDT work Matt was doing, but we need to get it put back together in a sane fashion, rather than chasing our tail like this.
Assignee: cnackers → dustin
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 11•12 years ago
|
||
Chris and I don't see any existing firewall configuration for Python on other, known-good w64 builders. Is this a known problem?
Assignee | ||
Comment 12•12 years ago
|
||
Kim's verifying right now that the hosts did OK in staging over the weekend.
Assignee: dustin → kmoir
Assignee | ||
Comment 13•12 years ago
|
||
This failed repeatedly on Aki's master over the weekend:
e:\builds\moz2_slave\m-cen-w64-ntly\build\toolkit\mozapps\installer\packager.mk:884:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py make-package-internal' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\config\rules.mk:582:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py libs' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\browser\build.mk:45:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py -C browser/installer' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\client.mk:205:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py -C obj-firefox package MOZ_PGO_INSTRUMENTED=1 MOZ_INTERNAL_SIGNING_FORMAT= MOZ_EXTERNAL_SIGNING_FORMAT=' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\client.mk:160:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py -f e:/builds/moz2_slave/m-cen-w64-ntly/build/client.mk profiledbuild' failed, return code 2
no data yet on what that means..
Assignee | ||
Updated•12 years ago
|
Assignee: kmoir → dustin
Comment 14•12 years ago
|
||
coop: The things that we tried yesterday seemed to have failed as well. We're blocked on this until we can get someone from releng who can help us define what this should look like and how it should work.
Flags: needinfo?(coop)
Comment 15•12 years ago
|
||
(In reply to Amy Rich [:arich] [:arr] from comment #14)
> coop: The things that we tried yesterday seemed to have failed as well.
> We're blocked on this until we can get someone from releng who can help us
> define what this should look like and how it should work.
Is the failure the same as comment #13?
I would note that the timing of the failures on Aki's test master coincides with bug 832992. Have we run another test series to rule that out?
Has anyone logged into the machine after/during the build to check for errant dialog boxes?
Flags: needinfo?(coop)
Assignee | ||
Comment 16•12 years ago
|
||
Those are all questions I can't answer (except the errant dialog boxes - see comment 10). Kim?
Updated•12 years ago
|
Flags: needinfo?(kmoir)
Comment 17•12 years ago
|
||
I setup my dev-master to run builds on these machines yesterday but they are still red so I'm still investigating :-) Will update when I know more.
Flags: needinfo?(kmoir)
Comment 18•12 years ago
|
||
I seem to have the same errors on these machines as they builds that are currently running in production
For instance, a WINNT 6.1 x86-64 mozilla-central build is failing make_pkg
e:\builds\moz2_slave\m-cen-w64\build\toolkit\mozapps\installer\packager.mk:588:0: command 'e:/builds/moz2_slave/m-cen-w64/build/obj-firefox/_virtualenv/Scripts/python.exe e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/pref -D_MSC_VER=1600 -DJAREXT= -DMOZ_ANGLE_RENDERER=1 -DMOZ_D3DX9_DLL=d3dx9_43.dll -DMOZ_D3DCOMPILER_DLL=D3DCompiler_43.dll -DMOZ_CHILD_PROCESS_NAME=plugin-container.exe -DMOZ_MSVC_REDIST=1600 -DMOZ_SHARED_MOZGLUE=1 -DMOZ_JSDEBUGGER -DNECKO_WIFI -DDLL_PREFIX= -DDLL_SUFFIX=.dll -DBIN_SUFFIX=.exe -DBINPATH=bin \
--format omni \
--removals e:/builds/moz2_slave/m-cen-w64/build/browser/installer/removed-files.in \
\
\
--jarlogs e:/builds/moz2_slave/m-cen-w64/build/obj-firefox/browser/installer/../../jarlog//en-US \
--optimizejars \
\
package-manifest ../../dist ../../dist/firefox' failed, return code 1
e:\builds\moz2_slave\m-cen-w64\build\toolkit\mozapps\installer\packager.mk:614:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64/build/build/pymake/pymake/../make.py make-package-internal' failed, return code 2
Traceback (most recent call last):
File "e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py", line 362, in <module>
main()
File "e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py", line 354, in main
args.source, gre_path, base)
File "e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py", line 142, in precompile_cache
errors.fatal('Error while running startup cache precompilation')
File "e:\builds\moz2_slave\m-cen-w64\build\python\mozbuild\mozpack\errors.py", line 101, in fatal
self._handle(self.FATAL, msg)
File "e:\builds\moz2_slave\m-cen-w64\build\python\mozbuild\mozpack\errors.py", line 96, in _handle
raise ErrorMessage(msg)
mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
e:\builds\moz2_slave\m-cen-w64\build\config\rules.mk:582:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64/build/build/pymake/pymake/../make.py libs' failed, return code 2
e:\builds\moz2_slave\m-cen-w64\build\browser\build.mk:45:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64/build/build/pymake/pymake/../make.py -C browser/installer' failed, return code 2
program finished with exit code 2
but the same error is occurring on builds running on non-staging slaves.
I don't see any errant dialog boxes on the machine.
Assignee | ||
Comment 19•12 years ago
|
||
OK, let's call it done! Do you want to toss those five (10,11,21,42,43, iirc) into production? Please open a new bug if you do find any further problems, or if the problem you've identified needs to be solved with changes to the image.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 20•12 years ago
|
||
(sorry, that's 10,20,21,42,43)
Comment 21•12 years ago
|
||
I didn't have time to run the other tests I wanted to run today because of other buildduty tasks, I'll look at this again on MOnday.
Comment 22•12 years ago
|
||
Additional tests were successful, win64-ix-slave11,20,21,42 and 43 are back in production.
Updated•11 years ago
|
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in
before you can comment on or make changes to this bug.
Description
•