This has been running for several days, with several builds, nightly mozilla builds on Linux/Ubuntu 21.10. Current one is 20220111093827.

STR:

1. Push to try
2. Open treeherder link on try, authenticate
3. Let the tab open waiting for completion
4. Come back a few hours later

Expected:
I'm still logged in

Actual:
I'm not logged in on Treeherder anymore.

Devtools console shows:

Could not renew login: 
Object { error: "login_required", errorDescription: "Login required", state: "xxx" }
AuthService.js:73:14
    _renewAuth AuthService.js:73

I need to clarify a few things. How do you know you're not logged in anymore after leaving the tab open for a few hours (besides looking at the console)? Do you try to perform a specific action and do you get an error message? Please try to provide as many details as possible.

(In reply to Sarah Clements [:sclements] from comment #1)

I need to clarify a few things. How do you know you're not logged in anymore after leaving the tab open for a few hours (besides looking at the console)? Do you try to perform a specific action and do you get an error message? Please try to provide as many details as possible.

Because:

• I have "Login / Register" in the top right corner
• Trying to retrigger tasks fails with the usual tc-level error showing I'm not authenticated

As I said and described, I'm not doing anything at all. Push to try, wait a few hours doing something else, come back to the tab to see, and kaboom, creds losts.

Might be similar to another bug 1646809. I'll try to look at this Monday but it might not be an easy fix.

Alexandre, has anything changed recently with regards to cookies being blocked or anything like that? Can you try opening Treeherder in private tab or window and see if the issue persists?

Also, next time this happens, please open the network tab and see if a call toapi/auth/login/ was made and what the response is.

(In reply to Sarah Clements [:sclements] from comment #4)

Alexandre, has anything changed recently with regards to cookies being blocked or anything like that? Can you try opening Treeherder in private tab or window and see if the issue persists?

No, nothing has changed to the best of my knowledge.

(In reply to Sarah Clements [:sclements] from comment #5)

Also, next time this happens, please open the network tab and see if a call toapi/auth/login/ was made and what the response is.

There was no api/auth/login call being made when:

Could not renew login: 
Object { error: "login_required", errorDescription: "Login required", state: "" }
​
error: "login_required"
​
errorDescription: "Login required"
​
state: ""
​
<prototype>: Object { … }
AuthService.js:73:14
    _renewAuth AuthService.js:73

(In reply to Sarah Clements [:sclements] from comment #4)

Alexandre, has anything changed recently with regards to cookies being blocked or anything like that? Can you try opening Treeherder in private tab or window and see if the issue persists?

Just reproduced in a private window. In both non-private and private, I have just noticed one non error message:

Le cookie « com.auth0.auth.STATE » a la politique « SameSite » définie sur « Lax » car son attribut « SameSite » n’est pas défini et « SameSite=Lax » est la valeur par défaut de cet attribut.

It states that the cookie com.auth0.STATE (with STATE being the value inside the error already mentionned) as a SameSite policy defined on Lax because its SameSite value is undefined and default is SameSite=Lax.

aha! It might be related to the same-site cookie issue. Which was just fixed and a new release published, so I'll update the auth0 library we're using and do some local testing.

Chris, can you help with this? I need to test a patch that a requires login.

Sure, what do you need?

(In reply to chris valaas [:cvalaas] from comment #11)

Sure, what do you need?

oops, wrong bug I need info'd you on :) It's bug 1751163 I need help with.

I have a pr that I need to test that I think will address the same site cookie issue, but I think this is a longstanding issue auth/login that might not even be specific to Treeherder.

(In reply to Sarah Clements [:sclements] from comment #13)

I have a pr that I need to test that I think will address the same site cookie issue, but I think this is a longstanding issue auth/login that might not even be specific to Treeherder.

Any news ?

Sorry, I've moved to another team and this fell off the radar due to other priorities. I have my pr on prototype... can you also login and see if the samesite site cookie issue happens on that deployment as well? https://prototype.treeherder.nonprod.cloudops.mozgcp.net/

(In reply to Sarah Clements [:sclements] from comment #15)

Sorry, I've moved to another team and this fell off the radar due to other priorities. I have my pr on prototype... can you also login and see if the samesite site cookie issue happens on that deployment as well? https://prototype.treeherder.nonprod.cloudops.mozgcp.net/

I'm unsure about SameSite issue, but I still get the same error about login required in the console on that deployment.

Yeah, I think that's part of the issues flagged in other bugs (added in "see also" section). And I still see the samesite cookie issue... there's still some ongoing issues with regards to the fix on the auth0 end, even with the v9.19.0, which I also tested: https://github.com/auth0/auth0.js/issues/1220#issuecomment-1024275244.

But I'll have to un-assign myself to this since I'm on the frontend team now.

So I cannot state why, but since a few days, it seems the issue has gone away. Is this a fix on treeherder side / its dependencies ? Is this a firefox-side fix ?

Since Sarah is not working on that anymore, maybe Joel you know if there was a recent change on treeherder side that could correlate ?

I would be surprised if this is treeherder side. We have done very little to change things. Any package upgrades in the last week or so have been related to development of treeherder. Maybe this is a browser thing?

This has started to happen again since a few days. Logged in this morning around 0700, it's 0910 and I had to login again.

Seeing this on a treeherder tab where I left debug console open

09:28:56,596
Comme certains cookies utilisent incorrectement l’attribut « SameSite », le fonctionnement ne sera pas celui attendu. 3
09:28:56,981 Le cookie « csrftoken » avec la valeur « Lax » ou « Strict » de l’attribut « SameSite » a été omis à cause d’une redirection intersite. ProxyChannelFilter.jsm:425:18
09:28:56,981 Le cookie « sessionid » avec la valeur « Lax » ou « Strict » de l’attribut « SameSite » a été omis à cause d’une redirection intersite. ProxyChannelFilter.jsm:425:18
09:28:57,311
Could not renew login: 
Object { error: "login_required", errorDescription: "Login required", state: "aau5tw0xjkolPhk6ayhZ4_Q0usZjZz9E" }
AuthService.js:73:14
    _renewAuth AuthService.js:73

Still happening.

I was suggested to remove all storage related to treeherder.mozilla.org within devtools Storage tab, and since I have not been able to reproduce the issue.

Expired authentication cookies are not purged fast enough in Firefox (bug 691973) and become too large in total to be sent completely.

Workaround:

1. Have a treeherder tab open.
2. Open the DevTools' Storage tab (Shift + F9):
3. On the left, expand "Cookies".
4. Click with the right mouse button onto the http://treeherder.mozilla.org item.
5. Choose Delete All.

Update: The expired cookies don't get sent - might hit a storage limit.

No more instance so far, after a full week.

Same, I cannot reproduce.

This is happening again. So there is something fishy. Not sure if treeherder is pushing storage too far or what.

I've started to once again experience this too.

Are you both seeing too large cookies?

(In reply to Marco Castelluccio [:marco] from comment #31)

Are you both seeing too large cookies?

How can I verify that?

You can follow the steps from comment 24 (stopping at step 3), the size is one of the columns. You could copy and paste the data here (filtering out the "value" column).

Having paid closer attention, what I'm seeing is not exactly the issue described in comment 0. Rather, it's when I open a new Treeherder tab that I find myself logged out, even if just logged in a few hours ago in a tab that I've since closed.

Cleaned up my cookies three days ago, and repro'd several times now.

Multiple (5) cookies named com.auth0.auth with size 152, one csrftoken of size 73 and one sessionid of size 41.

(In reply to Botond Ballo [:botond] from comment #34)

Having paid closer attention, what I'm seeing is not exactly the issue described in comment 0. Rather, it's when I open a new Treeherder tab that I find myself logged out, even if just logged in a few hours ago in a tab that I've since closed.

Update on this: I also lose my logged-in state if I refresh a Treeherder tab that's been open for a few hours.

I did check Cookies in the Storage tab in devtools prior to refreshing, and there was only one cookie with length 73, so it doesn't seem to be an issue of cookies getting too large.

This is still happening. Every 30 mins, dev console on an open treeherder tab shows a failure to renew login.

I've cleaned up cookies and local storage for both treeherder and TaskCluster, as well as logout and login back on both, to no luck.

Did you also try the steps to delete cookies from comment 24?

Are you using Treeherder in the default container or another container?

Are you able to reproduce in a clean Firefox profile?

(In reply to Marco Castelluccio [:marco] from comment #38)

Did you also try the steps to delete cookies from comment 24?

yes ...

Are you using Treeherder in the default container or another container?

non default, "Professional" where I host all mozilla-related stuff

Are you able to reproduce in a clean Firefox profile?

Running several profile in parallel is going to be really painful, can't we just add more debugging on treeherder to localize what is failing? We already have some error popping in console ...

(In reply to Alexandre LISSY :gerard-majax from comment #39)

Are you using Treeherder in the default container or another container?

non default, "Professional" where I host all mozilla-related stuff

This might be linked to why you're seeing this. I have seen similar issues on Taskcluster with a non-default container (bug 1456161). Botond, are you also using a non-default container?

Are you able to reproduce in a clean Firefox profile?

Running several profile in parallel is going to be really painful, can't we just add more debugging on treeherder to localize what is failing? We already have some error popping in console ...

Yes, unfortunately we have no experts of Auth0 in the team at the moment, so it won't be super quick. It's very likely this is a broader bug than Treeherder only BTW.

(In reply to Marco Castelluccio [:marco] from comment #40)

Botond, are you also using a non-default container?

Yes, Treeherder is configured to always open in my "Work" container.

OK, I think we have a pattern here. This is likely a container bug.

How can we track / debug this? Who knows about containers and cookies interactions ?

laxByDefault is turned off in Nightly (and elsewhere) so that's not the cause. I don't know why containers would make any difference unless your auth0 site was pinned to a different container, but you'd notice that because it would open up new tabs when it navigated there and back.

Multiple (5) cookies named com.auth0.auth with size 152, one csrftoken of size 73 and one sessionid of size 41.

you shouldn't be able to have multiple cookies of the same name in the same container. Does DevTools segregate cookies by origin attributes or does it show all cookies in all containers? The latter wouldn't be very useful

Redirecting needinfo, as I'm not able to reproduce this bug.

if the question is about the cookies display in devtools, you need to find someone knowledgeable of that.

I see Lax in the SameSite column ?

(In reply to Daniel Veditz [:dveditz] from comment #44)

I don't know why containers would make any difference unless your auth0 site was pinned to a different container, but you'd notice that because it would open up new tabs when it navigated there and back.

Hmm, I do get a new tab opening whenever I log into Treeherder (occasionally more than one, but only one remains open). It's in the same container though.

Treeherder and Firefox-CI are both setup to open by default in the same container. So there should be no problem there. Except if there's some third party I dont know about ?

When you login, you go through auth0.

(In reply to Marco Castelluccio [:marco] from comment #50)

When you login, you go through auth0.

Right. As much as I can tell, it goes throuhg the container as well.

Weird. I had a new error showing in console "login_required: multi-factor authentication required", I guess it's Auth0 expiration (my Google Calendar Tab, which lives in the same container, was expired at the same time). I logged out from Firefox-CI as well as Treeherder, re-auth'd everywhere (not Auth0 since I re-auth on it for Google a few minutes before), and since that moment yesterday afternoon, I have not lost auth on treeherder.

So far there's been no identified cookie or container bug, and could well be auth0 doing the wrong thing. laxByDefault cookies may have been implicated in earlier messages, but those are no longer enabled by default.

If you think it's cookies, please turn on logging and capture the traffic from when it's working to when it starts failing.

Dan, since this is only happening with containers enabled and never happening with containers disabled, shouldn't we consider it a containers bug?

This has started again for me (since a few days).

Unfortunately there is nothing we can do in Treeherder, it is a Core bug.
Dan, I'm going to move this back to Core::DOM: Security since it's only reproducible with containers enabled and I assume that's the right component for containers bugs. If there's a better component for Containers bug, feel free to move it again.

i am reproducing this without container.
I am using 3 systems - one at the paris office, two at home.
using strict as tracking protection

re-upping:

If you think it's cookies, please turn on logging and capture the traffic from when it's working to when it starts failing.

For the people using Containers: are all the sites involved in the same container? Or are some "pinned" to a different container?

Does Auth0 have a feature where it might de-auth some cookie from the same IP address if there's an auth in a different container? off-hand I'd say no 1) that doesn't make sense and 2) I commonly work with multiple auth0/container sessions without problems

(In reply to Daniel Veditz [:dveditz] from comment #58)

re-upping:

If you think it's cookies, please turn on logging and capture the traffic from when it's working to when it starts failing.

Redirecting needinfo to :gerard-majax and :botond, since I can't reproduce this myself.

(In reply to Daniel Veditz [:dveditz] from comment #59)

For the people using Containers: are all the sites involved in the same container? Or are some "pinned" to a different container?

I had pinned auth0 and others to "professional" tab and vs github was on "personal" but with the move to enforce SSO on github this makes me having to auth on both containers. I have not noticed an immediate problem due to this.

Does Auth0 have a feature where it might de-auth some cookie from the same IP address if there's an auth in a different container? off-hand I'd say no 1) that doesn't make sense and 2) I commonly work with multiple auth0/container sessions without problems

(In reply to Daniel Veditz [:dveditz] from comment #53)

So far there's been no identified cookie or container bug, and could well be auth0 doing the wrong thing. laxByDefault cookies may have been implicated in earlier messages, but those are no longer enabled by default.

If you think it's cookies, please turn on logging and capture the traffic from when it's working to when it starts failing.

Given we are talking about a behavior spanning potentially over days / weeks, is it really an actionable item ?

That being said, I'm away for a long period soon, so I dont think I can manage that kind of tracking before at best ~half of sept.

With a new profile and strict tracking protection enabled, the login persisted on Treeherder after 2+ hours.

Do all affected people have at least 2 mozilla.com SSO accounts?

Sylvestre does not use containers, but question for the others: The issue does not reproduce here with the following hosts assigned to the container used for Treeherder, and they are set to always open in such a container:

• firefox-ci-tc.services.mozilla.com
• login.taskcluster.net
• tools.taskcluster.net (likely not necessary for you)
• treeherder.mozilla.org

sso.mozilla.com is not assigned to any container but inherits the container environment of the opener.

Does anybody notice differences in their setup?

I dont know how to get an uptodate list of what is supposed to open into what, but I have at least forced firefox-ci-tc.services.mozilla.com as well as treeherder.mozilla.org to the "Professional" container.

ok, it clearly still repro: I logged out of treeherder and taskcluster right when I posted the previous comment, then connected again:

07:55:54,029 Could not renew login: 
Object { error: "login_required", errorDescription: "Login required", state: "u7IHhiY_BB27b8fyutKNiwMQrhB0HONP" }
AuthService.js:73:14
    _renewAuth AuthService.js:73

I've started logging using Cookies preset, and after 5 minutes it's already produced ~400MB of logs. Is this going to be actionable ? How am I supposed to share it with you ?

1.8GB of logs, re-auth'd at 0804 and first devtool console error reported 0819

Error popping circa 0819 in the console, and this in the logs:

2023-07-27 06:19:15.489146 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.489149 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/runtime.508cf28a.js
2023-07-27 06:19:15.489154 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.489156 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled
2023-07-27 06:19:15.489158 UTC - [Parent 3113884: Main Thread]: W/cookie
2023-07-27 06:19:15.491551 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.491554 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/397.ca13cc41.js
2023-07-27 06:19:15.491558 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.491560 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled
2023-07-27 06:19:15.491569 UTC - [Parent 3113884: Main Thread]: W/cookie
2023-07-27 06:19:15.502885 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.502888 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/index.0b2df09f.js
2023-07-27 06:19:15.502892 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.502895 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled

I should have shared it with you.

... since it's only reproducible with containers enabled and I assume that's the right component for containers bugs. If there's a better component for Containers bug, feel free to move it again.

That appears not to be true, but I can leave it here while we dig deeper. If it's basic Cookie stuff then "Networking: Cookies" is the right component, but the "rejected because cookies are disabled" errors in comment 67 are unexpected. Clearly cookies are not (globally) disabled or you wouldn't have been able to authenticate in the first place. Is there some anti-tracking heuristic that's kicking in? I could believe that there might be problems with auto-granting Storage Access to 3rd parties, but I thought Total Cookie Protection isolated (double-keyed) 3rd party cookies, not disabled them. Also, the grants in the link above are for an overly-generous 30 days, not just hours.

Paul: any ideas here?

I just checked on my machine and I'm observing the sessionid cookie being set with an Expires 2 hours in the future by https://treeherder.mozilla.org/api/auth/login. This would explain being logged out after a few hours of inactivity, right?

(In reply to :gerard-majax [PTO 01/08/2023-10/09/2023] from comment #67)

Error popping circa 0819 in the console, and this in the logs:

2023-07-27 06:19:15.489146 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.489149 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/runtime.508cf28a.js
2023-07-27 06:19:15.489154 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.489156 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled
2023-07-27 06:19:15.489158 UTC - [Parent 3113884: Main Thread]: W/cookie
2023-07-27 06:19:15.491551 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.491554 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/397.ca13cc41.js
2023-07-27 06:19:15.491558 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.491560 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled
2023-07-27 06:19:15.491569 UTC - [Parent 3113884: Main Thread]: W/cookie
2023-07-27 06:19:15.502885 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.502888 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/index.0b2df09f.js
2023-07-27 06:19:15.502892 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.502895 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled

These messages seem odd indeed given that your client seems to accept cookies prior to that? Do you perhaps have some special cookie permissions set? You can check under about:preferences -> Cookies and Site Data -> Manage Exceptions
Also, what is your cookie behavior set to? The pref is network.cookie.cookieBehavior.

(In reply to Paul Zühlcke [:pbz] from comment #71)

(In reply to :gerard-majax [PTO 01/08/2023-10/09/2023] from comment #67)

Error popping circa 0819 in the console, and this in the logs:

2023-07-27 06:19:15.489146 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.489149 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/runtime.508cf28a.js
2023-07-27 06:19:15.489154 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.489156 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled
2023-07-27 06:19:15.489158 UTC - [Parent 3113884: Main Thread]: W/cookie
2023-07-27 06:19:15.491551 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.491554 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/397.ca13cc41.js
2023-07-27 06:19:15.491558 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.491560 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled
2023-07-27 06:19:15.491569 UTC - [Parent 3113884: Main Thread]: W/cookie
2023-07-27 06:19:15.502885 UTC - [Parent 3113884: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2023-07-27 06:19:15.502888 UTC - [Parent 3113884: Main Thread]: W/cookie request URL: https://treeherder.mozilla.org/assets/index.0b2df09f.js
2023-07-27 06:19:15.502892 UTC - [Parent 3113884: Main Thread]: W/cookie current time: Thu Jul 27 06:19:15 2023 GMT
2023-07-27 06:19:15.502895 UTC - [Parent 3113884: Main Thread]: W/cookie rejected because cookies are disabled

These messages seem odd indeed given that your client seems to accept cookies prior to that? Do you perhaps have some special cookie permissions set? You can check under about:preferences -> Cookies and Site Data -> Manage Exceptions

None.

Also, what is your cookie behavior set to? The pref is network.cookie.cookieBehavior.

5, default value

FWIW the messages come from here https://searchfox.org/mozilla-central/rev/4044c34031d035fadb588143297ba5421419d44b/netwerk/cookie/CookieService.cpp#1762, which means there is a BEHAVIOR_REJECT present somewhere. This value is usually controlled by per-origin cookie permissions or the global cookie behavior. Which both doesn't seem to apply in your case...

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #62)

Do all affected people have at least 2 mozilla.com SSO accounts?

Just one in my case.

Sylvestre does not use containers, but question for the others: The issue does not reproduce here with the following hosts assigned to the container used for Treeherder, and they are set to always open in such a container:

• firefox-ci-tc.services.mozilla.com
• login.taskcluster.net
• tools.taskcluster.net (likely not necessary for you)
• treeherder.mozilla.org

sso.mozilla.com is not assigned to any container but inherits the container environment of the opener.

Does anybody notice differences in their setup?

I had sso.mozilla.com and treeherder.mozilla.org assigned to my Work container, but not the others. I assigned firefox-ci-tc.services.mozilla.com and taskcluster.net [note: the subdomains login.taskcluster.net and tools.taskcluster.net do not resolve for me, so I just assigned the primary domain] to my Work container as well, but that didn't seem to have any impact on the behaviour.

Same, i have only one SSO account (afaik).
Seems that Alexandre tried the cookie logging, so, i guess I don't need to do it :)

I logged in this morning (~10:30), and now (13:21) it's already lost session. I've already shared extensive debugging sessions, I am unsure what I can do more ...