Looking at: https://treeherder.mozilla.org/admin/auth/user/ We seem to just have is_staff, which is currently required in order to classify jobs and/or set up job visibility exclusion profiles (bug 1032152 will fix the former). However that same permission also allows access to the Django administration panel, where quite a lot of damage (either intentionally or accidentally) can be done - and quite a few people already have access, since is_staff had to be given out to people who just wanted to classify jobs or change visibility exclusion profiles. Also, 'is_staff' as a group name is likely to annoy non Moco contributors, since it makes the assumption that trusted contributors==staff. At the least I think we need the following groups: admin (replaces is_staff; can access the Django administration panel) sheriffing (can access job exclusion profiles) mozillians-vouched (can classify jobs and do other basic tasks) (or we could have even more groups, to say allow someone to manage project repository visibility exclusion profiles, but not for trunk repos etc) Once bug 1032152 is fixed, and we can remove most !sheriffs/admins from the is_staff group, this doesn't need to block IMO.

(In reply to Ed Morley [:edmorley UTC+0] from comment #0) > (or we could have even more groups, to say allow someone to manage project > repository visibility exclusion profiles, but not for trunk repos etc) Or allow people to add a new job type in the Django administration panel, but not add new users/give permissions to anyone else.

Since bug 1032152 is a blocker let's demote this, since short term once that's fixed we can just limit the list of people given the is_staff permission to {sheriffs, treeherder devs, some releng, ...} and so shouldn't have major problems.

(In reply to Ed Morley [:emorley] from comment #0) > mozillians-vouched (can classify jobs and do other basic tasks) Seems like we can: 1) Request an API key for mozillians.org 2) Use the API here to determine if a user is vouched: http://mozillians.readthedocs.org/en/latest/api/apiv2/api-users.html#examples I guess the question is: do we do this as part of user-creation, or every time that user wants to perform a more privileged action (eg triggering a job), or on a cron? I'm guessing if we can check their status on user creation (and still allow them to create an account if is_vouched is false, but it will just not have the vouched bit set in our DB) - and then run a cron every 24 hours to check users are still vouched, that should be fine?

+1 thanks Ed for raising the priority of this.

There's information below from dminor, dustin and smacleod. dustin wants to highlight that relengapi is the most attractive option. From dminor: ############ When I was working on Autoland to inbound back in the fall I did do LDAP checks in Autoland as pushing to Try is L1 and inbound is L3. The code I wrote is here: https://dxr.mozilla.org/hgcustom:version-control-tools/source/autoland/autoland/mozilla_ldap.py. I didn't have to worry getting the correct ldap account to check as I already had that from them pushing to Try. From dustin: ############ <armenzg> dustin, what is a way to know if a user has the right LDAP groups to at least to push to try? <armenzg> I want to see what is needed to add that to a service like TH <dustin> make an LDAP query for the posix group scm_level_1 <armenzg> or another tool <armenzg> where can we query? <armenzg> I want to see how complicated or easy it is <dustin> so if you have an LDAP bind account <dustin> you can connect to the LDAP servers (easiest if the service itself is hosted onsite, rather than in AWS) <dustin> and look up the group membership by email <dustin> scm_level_1 is a little weird because it's a POSIX membership <dustin> I had to do some special hacking for that in relengapi - https://github.com/mozilla/build-relengapi/pull/228 <armenzg> do you have some code where this is done? <dustin> yeah, see that pull and the files it modifies <armenzg> how hard would be for TH to reach these servers? <dustin> I think TH is in phx1? <armenzg> it seems they want to move TH to Heroku <dustin> yeah <dustin> that will make it harder, for sure <armenzg> do we have any service that is in a similar setup? <dustin> armenzg: not yet, but we've considered it with relengapi <dustin> you just need to do SSL client cert authentication to connect to LDAP <armenzg> what is SSL client cert authentication? <armenzg> what would be the host to use? <armenzg> should we file a bug for relengapi and wait for that? <dustin> armenzg: wait suddenly this is my project? :) <armenzg> dustin, you caught me! <armenzg> :) <armenzg> I want to put a plan forward <armenzg> no need for commitments <dustin> armenzg: so querying LDAP directly is one option <dustin> but LDAP and Persona are different things <dustin> and people often use different emails to identify themselves to those two things <dustin> it becomes something of a UX problem to make the two place nicely together <dustin> (e.g., if I login as dustin@mozilla.com, I'll have no LDAP groups) <philor> wonder whether we still have people like timeless with an LDAP account based on an email to which they have no access and thus cannot use it for persona even if they want one <dustin> philor: yeah, that's the kind of UX thing that comes up <dustin> armenzg: so if TH wants to use LDAP groups for permission, it should probably use LDAP for auth too <dustin> which can be either through Okta (which gets you a SAML assertion) or through HTTP Basic Auth (like intranet) <dustin> HTTP Basic Auth doesn't work outside of the Mozilla infrastructure, though (can't do the SSL client auth) <armenzg> dustin, if there was a way to match the personas to LDAP we would be good? <dustin> armenzg: yes, but there isn't such a way <dustin> so another option is to link TH auth to relengapi auth <armenzg> what would that do? <dustin> since relengapi already does this stuff <armenzg> delegate authentication to relengapi? <dustin> basically to "login" to treeherder you'd get redirected to relengapi, login there, and then authorize treeherder <armenzg> I understand that <dustin> similar to authorizing an app for github <ted> TC uses certs for auth internally, right? <dustin> yes <ted> the persona login is just a way to get a cert <armenzg> similar to what TC does with mach taskcluster-login command <ted> a temp cert <dustin> ted: yes, correct <dustin> armenzg: so as you can see, there are lots of options, none of them easy <dustin> relengapi even has a notion of permissions that's based on group membership, so you could just require certain relengapi permissions <armenzg> I will send this info to the TH team <dustin> ok <dustin> I'm happy to talk to them too <armenzg> sure! <armenzg> I will let them know From smacleod: ############## Here's how MozReview is handling the ldap problem, and how you could. MozReview: - When a user pushes to the special review repository with public key ssh, we take their ldap email (Which ssh puts in the USER env var) and store it in MozReview's database. - The pushing user is also providing MozReview credentials, so we are able to associate their MozReview account with this ldap email. - MozReview has a special ldap account it can use to query the ldap server for a users permissions (This user was requested in Bug 1168476) - When we would like to check an ldap users permissions, we query the ldap server using the code found at https://hg.mozilla.org/hgcustom/version-control-tools/file/e0129d27ef88/pylib/mozreview/mozreview/ldap/__init__.py#l38 The code linked above for querying scm_* group memberships is specific to that type of group - the way Mozilla's ldap works for different types of groups, so if you need to query non scm type groups it may have to be done differently - jabba on irc is very helpful with ldap stuff, and feel free to ask me questions. As for how you should handle LDAP, here is what I imagine: - Have a user created for you to query ldap (similar to Bug 1168476) - Assume that a user proving their email through persona login, is equivalent to them proving they have access to the ldap account associated with that email (This is the crux of the whole strategy - if a security review says this is not allowed we'd have to figure something else out - maybe we should talk to jabba about this. I know in the case of @mozilla.com email addresses it *is* true since you require your ldap credentials to login to persona) - Use code similar to what I've provided above to take the users email address and query ldap for the group permissions.

From jabba: ########### Over the internet, I expose ldap.mozilla.org:636 and ldap.mozilla.org:6363. The first one requires me to whitelist the source IP address, which if known is probably the easiest route. If that’s not a valid way to go like if you have unpredictable IPs, etc. then we can use the 2nd port, which doesn’t care about source IP, but it requires a client SSL certificate in order to connect. I don’t personally know how to do the client cert thing, but I believe the persona team is using this method and might be able to help if we go that route. Once you have connectivity, I can set up a bind user for tree herder to use, unless there is one already and it will need to structure a query to evaluate whether a given member is a member of a given group, I can probably help with that as needed.

From a conversation with jgriffin it seems that we don't want to implement this check in TH but create a servie that can do this. We also want to encourage users on TH to sign-in with their LDAP email and such service verifies if the user has the right scm level. If they do have the right privs, they can access these various actions we can take from TH. emorley: should this be a public API that TH hits after a persona login? Where in TH's code should we make these calls? I think that if relengapi could be such service it would be ideal.

scm_level_1 (having the ability to push to try) seems like the right level of check here, FWIW.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #10) > scm_level_1 (having the ability to push to try) seems like the right level > of check here, FWIW. It might not be; there are many gaia users who may want to retrigger jobs, but who don't have scm_level_1. I think simply having an LDAP account is probably sufficient here.

No, keep in mind that everyone has an LDAP account -- even 'guest', with the obvious password. You need to apply some kind of criterion.

Before I went on holidays, I believe I spoke with jgriffin about this again. There's no real need for yet another service. All the pieces is in place in this bug to talk to ldap and verify if an email address has LDAP associated to it. airmo does this already without needing to whitelist. (In reply to Dustin J. Mitchell [:dustin] from comment #12) > No, keep in mind that everyone has an LDAP account -- even 'guest', with the > obvious password. You need to apply some kind of criterion. What guest account are you referring to? Do we have such account publicly available? We could blacklist certain accounts if we need to. From mostlygeek: ################ Currently mozidp (the mozilla persona login for airmo) is a nodejs app that: - uses a client SSL cert to auth, since we don't have a fixed ip - looks up emails in ldap Here's the code: https://github.com/mozilla/vinz-clortho/ see: https://bugzilla.mozilla.org/show_bug.cgi?id=1024746 Also this is the latest code: https://github.com/mozilla-services/mozilla-idp/ And here is my commit that adds in client cert support: https://github.com/mozilla-services/mozilla-idp/commit/683fb57213a0294be6cddf7069ea2264235e50e0

You don't want a blacklist -- you need to specify exactly which users you want to have access. I was just pointing out that 'has an LDAP account' is roughly equivalent to 'has bothered to ask for an LDAP account' and isn't a meaningful attribute. Do you want employees? Contributors? scm level?

I think we want all of the above, basically everyone except "anonymous public user".

OK, I think the filter you want is (&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net))) then -- but Jabba will be able to confirm.

Hi jabba, Could you please let us know if what dustin proposed works for us? Thanks! (In reply to Jonathan Griffin (:jgriffin) from comment #15) > I think we want all of the above, basically everyone except "anonymous > public user". (In reply to Dustin J. Mitchell [:dustin] from comment #16) > OK, I think the filter you want is > (&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net))) then -- > but Jabba will be able to confirm.

This Thursday we would like to discuss this at the treeherder meeting and say what the plan will be.

(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #17) > Hi jabba, > Could you please let us know if what dustin proposed works for us? Thanks! > > (In reply to Jonathan Griffin (:jgriffin) from comment #15) > > I think we want all of the above, basically everyone except "anonymous > > public user". > > (In reply to Dustin J. Mitchell [:dustin] from comment #16) > > OK, I think the filter you want is > > (&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net))) then -- > > but Jabba will be able to confirm. Yeah, I think this will be good. It will limit access to human LDAP accounts (as opposed to service accounts) and allow any moco/mofo employees and any contributors/contractors/volunteers that have an LDAP account (which generally assumes NDA or committer's agreement or contract).

mdoglio and jonasfj are going to look into delegating authentication to TaskCluster auth.

After jonasfj, mdoglio and I met we believe there's no gain with the current model (we use Pulse to communicate between systems to take actions) to use TaskCluster for authentication. If we're only caring about people who have LDAP we believe OKTA will be all it is needed to implement on the Treeherder side. If the user directly hit the TaskCluster APIs it would make more sense to use TaskCluster as the auth method. mdoglio was to speak with the rest of Treeherder members to make sure this plan of actions works well.

dustin: do you know who is our point of contact for OKTA? I believe you mentioned that contributors could use OKTA. What is the process that a contributor in general would need to go through to have access through OKTA? Thanks

It's spelled "Okta", and Jabba is the best person to talk to.

Thanks dustin. Hi jabba, could you please have a look at comment 22? Thanks!

This is something we can investigate. We'd need the app to support different SAML endpoints for different permission levels and then we can assign the different endpoints to different groups of people in Okta. Unfortunately, in Okta, we pay per user licenses and we don't currently have any allocated for non-paid staff, although there are probably a few extra that we can use, depending on how many people we are talking about here?

(In reply to Justin Dow [:jabba] from comment #26) > Unfortunately, in > Okta, we pay per user licenses and we don't currently have any allocated for > non-paid staff, although there are probably a few extra that we can use, > depending on how many people we are talking about here? Pretty much "number of people who have scm_level_[1-3] minus the number of paid staff". Which I guess rules out Okta for us, hmm.

When I discussed this with Sean, for RelengAPI (which has about the same audience), I recall that we only pay for users who actually *use* the service. Which might be a much lower number (I estimated "a few" to start, scaling into "a few hundred" as adoption ramped up). I don't necessarily think this is prohibitive: I suspect we have a relatively small group of non-employees who are so engaged that they're clicking buttons on treeherder, as compared to the number of non-employees with LDAP or Bugzilla accounts. I think it's an easy case to make that Mozilla should support single sign-on for that group, as that goes a long way toward lowering the undesirable barrier to non-employee contribution.

I've happened to spot bug 1197381, which is pretty relevant in terms of choosing an approach here, if people hadn't already seen it. There's also more info in the links in bug 1197406. (Sorry closed bugs, at least for now).

For what its worth verifying email (using persona), and then look up email on mozillians.org using their new API is really easy. You can also check if email is member of a group on mozillians. So you could have a curated group called treeherder-powerusers that gets more permissions mozillians who are merely vouched. I know I would love to see curated mozillians groups for commit-level-1, -2 and -3. I think that would be useful other places too. The concept of using mozillians is also nice because everybody can see who has what access-level (inspecting LDAP groups isn't easy, I can't even see what permissions I have). (feel free to ping me for details if interested, it's not a perfect option - but decent)

Inspecting LDAP groups is pretty easy if you get a bind account. Could TreeHerder use tc-login?

(Marking this bug as moco-only temporarily so I can discuss the Persona EOL without having to be hand-wavy again. It will be announced publicly in a few weeks.) The Persona EOL is now due to be 30th November, see: https://groups.google.com/a/mozilla.com/forum/?hl=en#!topic/persona-eol/KRtX5RIcyZY As far as Treeherder is concerned this means: * We need to switch to an alternative form of authentication before then. * Whilst this switch will likely happen in another bug, the decision about which form of auth we should use will be influenced by our desire to give permissions to people other than those with moco email addresses in an automated manner (ie the goal of this bug). A WIP guide for Persona migration is at: https://mana.mozilla.org/wiki/display/Identity/Persona+migration+guide+for+internal+sites In it the alternatives for authentication are suggested as: (a) Firefox accounts (https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Firefox_Accounts/Introduction) (b) Okta (c) Rolling your own auth (which I guess in our case would be to just use Django's native auth) A quick recap of the Treeherder permissions requirements: (1) Both staff and non-staff users should be able to sign into Treeherder without having to request access in advance. (2) The non-staff group may likely be in the hundreds (since as time passes, Treeherder will require login for more and more features, eg persistent settings/views, "show my try pushes" etc). (3) Permissions will be used for features such as: job retriggering/cancellation, job scheduling (eg try extender), sheriff tasks (eg modifying which jobs are hidden by default). (4) We want to avoid having to manually grant users more permissions, by relying on other sources of truth. (This also means that it's easier for access to be revoked when people leave etc). (5) Whilst some of these permissions could be verified downstream (eg Treeherder only asserts that a Pulse retrigger request was made by user X, with it then being up to the consumer of the Pulse stream to check permissions for that email), there are at least some cases where Treeherder would need to make the determination itself (eg permission for modifying hidden jobs), which means we might as well handle everything ourselves anyway. (6) Whilst whitelisting *@mozilla.com and also querying LDAP for eg scm_level_X seems like it should cover everyone, we need to find out if there are many non-staff developers that don't have an LDAP account (eg gaia/other GitHub projects that report to Treeherder). If so, we may need to do things like querying GitHub org membership or Mozillians groups or ...? In my opinion Okta is not an option, since there are both licence cost limitations, but more importantly even if we're happy to hand accounts out in the hundreds, it will presumably not be self-service, and would require someone to file a bug to be able to sign into Treeherder (violating requirement [1]). There is support for Firefox Accounts in the django-allauth package (which is what MDN will be using as they switch away from Persona): https://github.com/pennersr/django-allauth/pull/744/files http://www.intenct.nl/projects/django-allauth/ Given that django-allauth offers a superset of the native Django auth, it makes sense to use that rather than the latter. django-allauth also allows people to sign in via GitHub (and even link both a GitHub and a 'Firefox Accounts' account at the same time). We could then use a combination of {matching *@mozilla.com, querying LDAP for that email to check groups, querying GitHub org membership} to determine permissions. Thoughts? :-)

It wfm, however, I assume it is more important for others more knowledgable to share their thoughts than me (as I don't know the implications of choosing one auth over other).

:peterbe started a thread on a somehow similar topic a month ago or so https://groups.google.com/d/msg/mozilla.dev.webdev/Vv-UwBYnCjs/0AC8D-VaAAAJ The tl;dr is that with FxAccounts you can't invalidate a user session (for example when an ldap account is closed) which was the case with persona. Peter seems to be willing to implement a separate service that we could use to circumvent the problem. I'll NI him to see if he has any suggestion.

Yes I intend to build such a service. It'll have two pieces of functionality. 1) You can query if an email is in LDAP, is an employee or belongs to a certain group. All these API queries will only return True or False. Never a list of emails or groups. 2) When the status of an LDAP entry changes it will send a HTTP POST to your URL endpoint of choice. You'll have to decipher, in the JSON payload, if the change was that the user changed employment status. I now know of 4 services that need this service. This one, Air Mozilla, Crash Stats, Taskcluster. There might be more. The work has not been planned "on paper" (e.g. Deliverables, my chat with my manager). Just gathering requirements and ideas. I do have a mental head-start in this space (having done the query part before) but I would love for this to be a project with multiple owners and drivers.

So Persona EOL is now public; opening the bug. There's now another option on the Mana page migration guide, which allows use of Okta for mozilla.com emails and fallback to FxA for everyone else; example project at: https://github.com/rfk/hellomo (In reply to Mauro Doglio [:mdoglio] from comment #34) > :peterbe started a thread on a somehow similar topic a month ago or so > https://groups.google.com/d/msg/mozilla.dev.webdev/Vv-UwBYnCjs/0AC8D-VaAAAJ > The tl;dr is that with FxAccounts you can't invalidate a user session (for > example when an ldap account is closed) which was the case with persona. Ah I hadn't seen that, thank you. (In reply to Peter Bengtsson [:peterbe] from comment #35) > Yes I intend to build such a service. It'll have two pieces of > functionality. > 1) You can query if an email is in LDAP, is an employee or belongs to a > certain group. All these API queries will only return True or False. Never a > list of emails or groups. > 2) When the status of an LDAP entry changes it will send a HTTP POST to your > URL endpoint of choice. You'll have to decipher, in the JSON payload, if the > change was that the user changed employment status. This sounds great! I agree that lots of services would like to make use of such functionality.

AMO's migration plan for Persona -> Firefox Accounts, in case it's of use later: https://github.com/mozilla/addons/blob/master/docs/ux/fxa-migration.rst

Will be interesting to see the outcome of: "Grant another site some of your scopes (via an OAuth-like flow)" ...from: http://code.v.igoro.us/posts/2016/02/taskcluster-login-ldap.html :-)

I had treeherder in mind when I wrote that :)

(In reply to Ed Morley [:emorley] from comment #39) > Will be interesting to see the outcome of: > "Grant another site some of your scopes (via an OAuth-like flow)" > ...from: > http://code.v.igoro.us/posts/2016/02/taskcluster-login-ldap.html > > :-) (In reply to Dustin J. Mitchell [:dustin] from comment #40) > I had treeherder in mind when I wrote that :) Hi Dustin - is that feature available currently, or still a TODO? I've read: https://docs.taskcluster.net/manual/apis https://docs.taskcluster.net/reference/platform/auth/api-docs etc ...but didn't see anything we could use? Thanks :-)

Right now, you'd need to redirect to login.taskcluster.net, as described here: https://github.com/taskcluster/taskcluster-login#access-grant-for-tools We can add some sites to the whitelist there so that users don't need to type the site name. Note that this does not allow `http://` sites; as such, you'll need to use something like ngrok for local development.

..to finish that thought, we are planning to move that UI to tools.taskcluster.net someday, but when we do so we'll keep backward compatibility with the login redirect URL, so you won't need to change anything.

This bug covers several interconnected but separate issues. I've split these up into new bugs with hopefully clearer descriptions, now that we have a rough plan: 1) Bug 1273034 - Transition Treeherder login from Persona to login.taskcluster.net -> This is just about switching the login/SSO solution and won't make any permissions changes. This will be worked on by me this quarter. 2) Bug 1273096 - Use the accessToken generated by login.taskcluster.net to interact with Taskcluster APIs -> Covers what this bug was originally filed for: coming up with a less hacky way to interact with Taskcluster APIs, to replace the current *@mozilla.com whitelist. Depends on bug 1273034. Out of scope for me this quarter. 3) Bug 1273092 - Use login.taskcluster.net's LDAP-derived scopes to set internal Treeherder permissions -> Using LDAP groups instead of manually giving people Treeherder sheriffing panel permissions via the Django admin panel. Depends on bug 1273034. Not urgent since the list of users who have such permissions is small and the current solution is working ok.