As seen on bug 1320642 we fail to load pages which are served through the public IP of the local host. Marionette tests will hang in navigate() for about 300s for each and every call to a locally hosted page. Given that we have a hard build timeout of 60 minutes, Jenkins kills the job after that time.

All Linux and OS X machines for staging have been updated and as it looks like we do not have a problem anymore. I will follow-up with production soon.

Looks like there are still problems with Linux not being able to load some HTTPS URLs.

I updated the ESX pages for the no-proxy settings of 10.0.0.0/24. It at least make nearly all tests passing. But i still see failures for the safebrowsing tests when the following page gets loaded: https://fr.phish-error.mozilla.com/?hl=fr&url=https%3A%2F%2Fwww.itisatrap.org%2Ffirefox%2Fits-a-trap.html Here the browser hangs and doesn't load that page. I will investigate more tomorrow.

I was playing around with the wpad.dat for automatic proxy configuration and it seems to work fine except that two tests are failing which want to verify the ev certificate via the certificate's OSCP URL. This check fails because loading ocsp.digicert.com never ends. As it looks like this is an issue with the automatic proxy configuration. :/ I will check if setting all the proxy values for gnome similar to the env variables will fix it.

Ok, so with the following settings all is working fine now: export DISPLAY=:0 gsettings reset-recursively org.gnome.system.proxy gsettings set org.gnome.system.proxy use-same-proxy true gsettings set org.gnome.system.proxy mode "'manual'" gsettings set org.gnome.system.proxy ignore-hosts "['localhost', '127.0.0.0/8', '10.22.73.0/8', '*.scl3.mozilla.com', '*.phx1.mozilla.com', 'mm-ci-staging', 'mm-ci-production', '*.mozqa.com']" gsettings set org.gnome.system.proxy.http host "proxy.dmz.scl3.mozilla.com" gsettings set org.gnome.system.proxy.http port 3128 gsettings set org.gnome.system.proxy.http use-authentication false gsettings set org.gnome.system.proxy.http enabled true gsettings list-recursively org.gnome.system.proxy I will let this run today and update production later or tomorrow.

So bug 1322347 has been fixed faster than expected. I will wait with the push to production and maybe enable automatic-proxy detection instead. So production systems have to wait.

All Linux and Mac slaves in staging are looking good. Going to update production today.

This has been finished for production now. Everything seems to be green again.