https://gecko.readthedocs.org/ frequently fails to build. We keep hitting apparent scaling limits with ReadTheDocs. Plus the setup is a bit fragile. As of bug 1048650 we now generate the Sphinx docs as part of TaskCluster. So it makes sense to publish these generated docs somewhere and to tie that to TaskCluster. Bug 877197 is on file for MDN to support publishing Sphinx docs. But with the recent pivot of MDN, my guess is that bug is going nowhere and will be WONTFIX'd. We should come up with somewhere else to publish the Firefox Sphinx docs. Somewhere reliable. Preferably somewhere hosted or under control of Mozilla so we can fix it when it is broken. Apparently you can publish TC artifacts to S3 buckets. You can also configure S3 buckets for web site publishing. We should be able to create an S3 bucket, do some DNS and x509 certificate magic, and have TC publish the Sphinx docs. Dustin: is TC in a position to manage the S3 bucket? Or do I need to work the EC2 bits on my end?

We can set up publishing to an S3 bucket. It's a little fiddly on the backend, and has the restriction that the TC AWS account has to own the bucket, but otherwise is pretty straightforward. We can also set up CloudFront to use custom DNS and TLS (if desired). The uploading task basically calls a TC endpoint and gets back some AWS STS credentials which it can then use to upload.

Well, since the TC AWS account has to own the bucket, I guess I'm off the hook for implementing this :) Seriously though, I can help with Sphinx updates if needed. As far as prioritization, this likely isn't at the top of anyone's list. That being said, MDN is effectively going away for build system docs. And sphinx-js is a nifty way to document JS which I'd like to explore but fear straining RTD. I suspect if we made the in-tree Sphinx docs easier to update and more useful, a lot of people would be very appreciative. Since RTD inconsistency is undermining trust in the in-tree Sphinx docs, having this implemented and having stable Sphinx docs generation would likely spur interest in improving docs.

OK, I'll get a task created that uploads a hello.html, and then let you take it from there.

That division of labor is very acceptable to me!

Actually, https://docs.taskcluster.net/manual/using/s3-uploads means that any user can own the bucket. I'll set it up with a TC bucket, but that's easy to change later.

I created bucket gecko-docs.mozilla.org, with the following bucket policy: { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadForGetBucketObjects", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::gecko-docs.mozilla.org/*" } ] } And the result is http://gecko-docs.mozilla.org.s3-website-us-west-2.amazonaws.com/ Once we figure out a hostname and get a TLS cert for it, we can set up cloudfront. I have a bit more work to do to get credentials set up and run this in automation, then I'll post a patch.

The gecko-docs-upload IAM user has this policy (also named gecko-docs-upload) applied: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::gecko-docs.mozilla.org" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::gecko-docs.mozilla.org/*" ] } ] }

I should add I didn't test the task invocation, so the requests/session stuff might be wrong.

Comment on attachment 8891500 [details] Bug 1382729: Upload documentation to S3 automatically; https://reviewboard.mozilla.org/r/162626/#review168020 ::: tools/docs/mach_commands.py:113 (Diff revision 1) > + @Command('doc-upload', category='devenv', > + description='Generate and upload documentation from the tree.') There are already too many mach commands. Please file a follow-up to consolidate `mach doc` into a sub-command based command. ::: tools/docs/mach_commands.py:126 (Diff revision 1) > + def upload_docs(self, bucket, region, what=None): > + self._activate_virtualenv() > + self.virtualenv_manager.install_pip_package('boto3==1.4.4') > + > + outdir = os.path.join(self.topobjdir, 'docs') > + #self.build_docs(what=what, outdir=outdir, format='html') I think you forgot to uncomment this :) ::: tools/docs/mach_commands.py:173 (Diff revision 1) > + if content_type: > + extra_args['ContentType'] = content_type > + if content_encoding: > + extra_args['ContentEncoding'] = content_encoding > + print('uploading', pathname) > + s3.upload_file(pathname, bucket, pathname, ExtraArgs=extra_args) This will be much faster if you use a thread pool with futures like we do for the decision task. But it probably isn't worth doing initially.

Yeah, I only promised I'd get it working, remember? :) A few other ideas for improvement: - list the bucket contents and only upload changed files - list the bucket contents and ensure that any objects already present are either present in the updated docs, or have a redirect defined (so that a link never dies)

I'll land this with the new job set to tier 3 so I can see it fail..

https://hg.mozilla.org/integration/mozilla-inbound/rev/19f92c04608cec275dab73e8acad5141de8a5c44 Bug 1382729: Upload documentation to S3 automatically; r=gps

This is causing https://public-artifacts.taskcluster.net/bxhzSaMeT42advEOxbWW2w/0/public/logs/live_backing.log [task 2017-07-31T15:45:04.609938Z] Creating task with taskId boA9lTVRSiSgsKsw-bo8ig for source-test-doc-upload [task 2017-07-31T15:45:04.611541Z] Starting new HTTP connection (4): taskcluster [task 2017-07-31T15:45:04.612161Z] Starting new HTTP connection (5): taskcluster [task 2017-07-31T15:45:04.997049Z] "PUT /queue/v1/task/boA9lTVRSiSgsKsw-bo8ig HTTP/1.1" 403 11670 [task 2017-07-31T15:45:04.998205Z] You do not have sufficient scopes. This request requires you [task 2017-07-31T15:45:04.998257Z] to have one of the following sets of scopes: [task 2017-07-31T15:45:04.998362Z] [ [task 2017-07-31T15:45:04.998442Z] [ [task 2017-07-31T15:45:04.998557Z] "secrets:get:project/releng/gecko/build/level-3/gecko-docs-upload", [task 2017-07-31T15:45:04.998683Z] "secrets:get:project/taskcluster/gecko/hgfingerprint", [task 2017-07-31T15:45:04.998741Z] "docker-worker:cache:level-1-checkouts-v1", [task 2017-07-31T15:45:04.998793Z] "queue:route:tc-treeherder.v2.try.bcdd7a362ce11f0a20779c27d9b9d7261d853f2e.209370", [task 2017-07-31T15:45:04.998872Z] "queue:route:tc-treeherder-stage.v2.try.bcdd7a362ce11f0a20779c27d9b9d7261d853f2e.209370" [task 2017-07-31T15:45:04.998895Z] ] [task 2017-07-31T15:45:04.998913Z] ] on try. I don't know why the job is running on try :(

backed out in https://hg.mozilla.org/mozilla-central/rev/1be0c1da06076f85c69cd8a9d244e0164ec544d9 And ahal points out, run-on-projects doesn't apply to `-j`obs :(

Comment on attachment 8892987 [details] Bug 1382729: add support for run-on-projects with -j; https://reviewboard.mozilla.org/r/163990/#review169296 In my experimentation this increased the number of jobs filtered by target_tasks by one, and the source-test-doc-upload task did not appear in the output afterward.

Comment on attachment 8891500 [details] Bug 1382729: Upload documentation to S3 automatically; https://reviewboard.mozilla.org/r/162626/#review169300

Comment on attachment 8892987 [details] Bug 1382729: add support for run-on-projects with -j; https://reviewboard.mozilla.org/r/163990/#review169302 What could go wrong.

Well, I can imagine 10,000 things, so it's probably none of those :) I ran a try job and it didn't fail, so that's a start!

Pushed by dmitchell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d2ad1e533a35 Upload documentation to S3 automatically; r=gps https://hg.mozilla.org/integration/autoland/rev/f0df47303111 add support for run-on-projects with -j; r=gps

https://hg.mozilla.org/mozilla-central/rev/d2ad1e533a35 https://hg.mozilla.org/mozilla-central/rev/f0df47303111

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/mozilla-central/rev/36ad88e6b7b2 Call proper mach command to uploads docs; r=dustin, a=bustage

Still busted on central with: HTTPError: 404 Client Error: Not Found for url: http://taskcluster/secrets/repo:hg.mozilla.org/mozilla-central/gecko-docs-upload It sounds like Dustin knows what's wrong.

I've set up gecko-docs.mozilla.org-{l1,l2} with 1-day expiration lifecycle rules, so the data won't stick around. I also set up gecko-upload-docs-{l1,l2} IAM users with policies like { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::gecko-docs.mozilla.org-l1" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::gecko-docs.mozilla.org-l1/*" ] } ] } I set up secrets containing creds for those users. Now to update the task.

This should fix things up. This succeeded in try (finally): https://tools.taskcluster.net/groups/N2XKmyNZTHSgyfstzhi7yA/tasks/XVevcptvS2C8fnOWMIDn5Q/runs/0/logs/public%2Flogs%2Flive.log so i expect it will do OK at level 3 too.

By giving Try tasks the ability to upload to an S3 bucket, we've just opened up the potential for hosting abuse. Also, if we screw things up, mistakes could persist forever. I assume there is a bucket policy with aggressive auto expiration to prevent bad things?

https://hg.mozilla.org/integration/mozilla-inbound/rev/ccc4663a6c255cd70bace1e4680fee660505f316 Bug 1382729: allow uploading docs at all levels; r=gps

https://hg.mozilla.org/projects/date/rev/d2ad1e533a35ae3944bd46c6098d54ba2aa9ac02 Bug 1382729: Upload documentation to S3 automatically; r=gps https://hg.mozilla.org/projects/date/rev/f0df473031115857ccb0ddf96b14b34780bcf233 Bug 1382729: add support for run-on-projects with -j; r=gps https://hg.mozilla.org/projects/date/rev/36ad88e6b7b248c2f2ae59b80477e5474dd653dc Bug 1382729 - Call proper mach command to uploads docs; r=dustin, a=bustage

(In reply to Gregory Szorc [:gps] from comment #36) > By giving Try tasks the ability to upload to an S3 bucket, we've just opened > up the potential for hosting abuse. Also, if we screw things up, mistakes > could persist forever. I assume there is a bucket policy with aggressive > auto expiration to prevent bad things? Yes, sorry I was thinking that discussion was in this bug. It's a private bucket, without website support enabled, and it has a 1-day expiration set. So no big risk.

Followup in other bugs..

Removing leave-open keyword from resolved bugs, per :sylvestre.