If I pull all sources from NSS_3_11_BRANCH, build and run cert.sh, it fails : cert.sh: Creating an EC CA Certificate TestCA-ec ========================== /h/monstre/export/home/nss/311/mozilla/tests_results/security/bigbelly.1/CA cert.sh: Creating EC CA Cert TestCA-ec -------------------------- certutil -s "CN=NSS Test CA (ECC), O=BOGUS NSS, L=Mountain View, ST=California, C=US" -S -n TestCA-ec -k ec -q secp521r1 -t CTu,CTu,CTu -v 600 -x -d . -1 -2 -5 -f ../tests.pw.2858 -z ../tests_noise.2858 -m 1 Generating key. This may take a few moments... certutil: signing of data failed: security library: invalid algorithm. cert.sh ERROR: Creating EC CA Cert TestCA-ec failed 255 return value is 255 cert.sh: Exit: 6 Fatal - failed to create EC CA cert I know we are doing some builds that includes some files from other locations than the branch, but that doesn't excuse the fact that the QA on the branch should be able to run.

Julien, you found a problem today where a test was getting certutil from a different directory in the PATH. Does that explain this bug?

Yes, it was a PATH problem. I wish our test scripts would check that the binaries being tested actually exist on the system before trying to run them. It could enforce it simply by running them explicitly from the mozilla/dist location. I'm changing this bug to reflect that request.

To make things clearer, the values of my BUILD_OPT and USE_64 in my shell running all.sh didn't match with the values of my BUILD_OPT and USE_64 in my build. The only reason all.sh went as far as it did is that it picked up certutil from Solaris in /usr/sfw/bin, which is my user's PATH . Of course those versions don't have ECC in them, so it aborted with the "invalid algorithm" error. This PATH issue is a serious problem. We could end up running all.sh and get results from the Solaris bits, instead of the bits we think we are testing. Eventually, the Solaris bits will have ECC in them too, and we won't be able to detect this PATH problem - or there may be a different failure mode. The issue is very treacherous, because there is no indication that the bits are being pulled from Solaris instead of mozilla/dist when we run all.sh . The QA script needs to make sure that the bits it wants to run actually have been built. It also wouldn't be a bad idea to run the tools with their fully-qualified pathnames - eg run "$(DIST)/$(OBJDIR)/bin/certutil" instead of just "certutil".

Comment on attachment 297153 [details] [diff] [review] Patch. Slavo, This patch no longer applies for memleak.sh . Did you try it on windows ?

Comment on attachment 297153 [details] [diff] [review] Patch. This is fine, as long as you resolve the merge conflict for memleak.sh.

Checking in cert/cert.sh; /cvsroot/mozilla/security/nss/tests/cert/cert.sh,v <-- cert.sh new revision: 1.50; previous revision: 1.49 done Checking in cipher/cipher.sh; /cvsroot/mozilla/security/nss/tests/cipher/cipher.sh,v <-- cipher.sh new revision: 1.15; previous revision: 1.14 done Checking in cipher/performance.sh; /cvsroot/mozilla/security/nss/tests/cipher/performance.sh,v <-- performance.sh new revision: 1.6; previous revision: 1.5 done Checking in common/init.sh; /cvsroot/mozilla/security/nss/tests/common/init.sh,v <-- init.sh new revision: 1.60; previous revision: 1.59 done Checking in crmf/crmf.sh; /cvsroot/mozilla/security/nss/tests/crmf/crmf.sh,v <-- crmf.sh new revision: 1.2; previous revision: 1.1 done Checking in dbtests/dbtests.sh; /cvsroot/mozilla/security/nss/tests/dbtests/dbtests.sh,v <-- dbtests.sh new revision: 1.15; previous revision: 1.14 done Checking in dbupgrade/dbupgrade.sh; /cvsroot/mozilla/security/nss/tests/dbupgrade/dbupgrade.sh,v <-- dbupgrade.sh new revision: 1.2; previous revision: 1.1 done Checking in fips/fips.sh; /cvsroot/mozilla/security/nss/tests/fips/fips.sh,v <-- fips.sh new revision: 1.32; previous revision: 1.31 done Checking in iopr/cert_iopr.sh; /cvsroot/mozilla/security/nss/tests/iopr/cert_iopr.sh,v <-- cert_iopr.sh new revision: 1.6; previous revision: 1.5 done Checking in iopr/ocsp_iopr.sh; /cvsroot/mozilla/security/nss/tests/iopr/ocsp_iopr.sh,v <-- ocsp_iopr.sh new revision: 1.7; previous revision: 1.6 done Checking in iopr/ssl_iopr.sh; /cvsroot/mozilla/security/nss/tests/iopr/ssl_iopr.sh,v <-- ssl_iopr.sh new revision: 1.5; previous revision: 1.4 done Checking in memleak/memleak.sh; /cvsroot/mozilla/security/nss/tests/memleak/memleak.sh,v <-- memleak.sh new revision: 1.26; previous revision: 1.25 done Checking in perf/perf.sh; /cvsroot/mozilla/security/nss/tests/perf/perf.sh,v <-- perf.sh new revision: 1.6; previous revision: 1.5 done Checking in pkits/pkits.sh; /cvsroot/mozilla/security/nss/tests/pkits/pkits.sh,v <-- pkits.sh new revision: 1.18; previous revision: 1.17 done Checking in sdr/sdr.sh; /cvsroot/mozilla/security/nss/tests/sdr/sdr.sh,v <-- sdr.sh new revision: 1.7; previous revision: 1.6 done Checking in smime/smime.sh; /cvsroot/mozilla/security/nss/tests/smime/smime.sh,v <-- smime.sh new revision: 1.28; previous revision: 1.27 done Checking in ssl/ssl.sh; /cvsroot/mozilla/security/nss/tests/ssl/ssl.sh,v <-- ssl.sh new revision: 1.88; previous revision: 1.87 done Checking in ssl/ssl_dist_stress.sh; /cvsroot/mozilla/security/nss/tests/ssl/ssl_dist_stress.sh,v <-- ssl_dist_stress.sh new revision: 1.10; previous revision: 1.9 done Checking in tools/tools.sh; /cvsroot/mozilla/security/nss/tests/tools/tools.sh,v <-- tools.sh new revision: 1.15; previous revision: 1.14 done

Reopening. Slavo, ever since your checkin for this bug the tinderbox has been orange for host "securitytip memleak Linux dopushups". Please determine the reason that the tree is orange.

Previous change caused that Valgrind parser in memleak.sh script didn't work correctly - stacks containing absolute paths didn't match ignored patterns. This patch should fix the problem (cuts path and uses only binary).

Checking in memleak.sh; /cvsroot/mozilla/security/nss/tests/memleak/memleak.sh,v <-- memleak.sh new revision: 1.27; previous revision: 1.26 done