Hello

I reported to the "DownThemAll" Extension Github because i found this issue using it, but later i tested the "Save In.." Extension and i have the same result (with a bit more information) so maybe is more related with WebExtension API itself than the concrete extensions.

Summarizing, since a month ago, i can't use an extension to download an image from DeviantArt, but i can use the normal download system in Firefox. Using
"DownThemAll" it only fails the download, and using "Save In..." it also popups a "SERVER_BAD_CONTENT" message.

I will copy the issue writed for "DownThemAll" posted here https://github.com/downthemall/downthemall/issues/214 explaining everything, with examples and additional context of the website, but with "Save In..." is the same (with the "SERVER_BAD_CONTENT" message).

Describe the bug

Until this week, i was using DownThemAll to download images from DeviantArt and >choose different folders on the fly. This week began to fail the downloads (I think it puts >"Not Found"). Note that a normal download with Firefox works.

Steps to reproduce the behavior:

You must be logged in DeviantArt to download images and using the new site "Eclipse".
Go to https://www.deviantart.com/sandara/art/Hydra-2-828769182
Secondary Click on "Download" (Arrow going down icon below the image) and choose >"Save link with OneClick"

Expected behavior

It downloads the file without any problem.
It download with a "readable" filename (hydra_2_by_sandara_ddpfee6.jpg).

Additional context

DeviantArt has been doing a lot of changes recently, but always did something with the filenames. For example when you use OneClick to download the example file, in the list you can see the initial filename is ddpfee6-011a5b5f-386b-40a8-849e-7f28a9dfb5dd.jpg, it changes to hydra_2_by_sandara_ddpfee6.jpg but now when it fails, appears the initial name again as the failed download. When it was working, the download filename was always the "readable" one hydra_2_by_sandara_ddpfee6.jpg with the title and author what is what i want.
The URL for download (now) is https://www.deviantart.com/download/828769182/ddpfee6-011a5b5f-386b-40a8-849e-7f28a9dfb5dd.jpg?token=42235a27d93330ff35d10f73211ea8b8bd4495e6&ts=1582198546 and there is something with the "token" because if you try to download the file a few minutes later than the page was opened, a normal download fails because it doesn't find the file (maybe is related of what is happening with DownThemAll).

That filename thing is a bit of a problem because in the past with the old site you could save all images directly (not using the "Download" link) and they always mainteined the readable filenames (and also appeared in a web search with that filenames). Now sometimes happens, sometimes don't, but with the Download link (if it's allowed by the author) always appears the readable filename. This is not a fault of DownThemAll, it's only to give some context about the filenames, for sure is something they are trying to protect the images (I think is worse in my opinion, but that's not the point of this issue).

I can post more info (console?) if you say me where to find relevant info about this issue.

I'm using Firefox 73.0.1 (64 bit).

I can try to gather more information if it's needed.

Thanks.

Hi Nils, this seems to be related to your extension and a change of yours. Could you take a look? Thanks!

Pointing out that this is now happening to me also with images sent via Twitter DMs, with the Twitter Web and Tweetdeck.

The only difference is with "Save Image in Folder" via Twitter Web, i used it and it does nothing, it doesn't say "fail" download or anything (maybe related of what are the filenames gathered with the web version). But "Save In" and "DownThemAll" fails exactly the same as they does with DeviantArt ("Save In" shows the "SERVER_BAD_CONTENT" message also). With Tweetdeck, all of them ("DownThemAll", "Save In" and "Save Imagen in Folder" fails exactly the same as they does with DeviantArt images.

Stock Firefox "Save Image" download all images via Tweetdeck or the Twitter Web without problem.

To note, i use "Better Tweetdeck" extension (Maybe that's why the difference noted above, it gathers a different resolution image or something like that)

Could you provide the exact reproduction steps so that anyone who starts out with an empty Firefox profile, and no familiarity with Twitter or DeviantArt can try to reproduce and verify the issue?

Well... ok.

Initial requeriments:

• Firefox 75.0 64bit (My test version, should be reproducible in others)
• Clean Profile
• Install these extensions (maybe others have the same issues, but i only can talk about these) to test the issue with all of them.
  • DownThemAll! https://addons.mozilla.org/es/firefox/addon/downthemall/ (Default Settings)
  • Save Image In Folder https://addons.mozilla.org/es/firefox/addon/save-image-in-folder-ex/ (Default Settings)
  • Save In... https://addons.mozilla.org/es/firefox/addon/save-in/ (Default Settings)
• A Twitter.com account, logged in (For looking Direct Messages (DMs) in Tweetdeck and Twitter Web)
• A DeviantArt.com account, logged in (You can't use the download button without being logged)

Steps to Reproduce the issue:

• For DeviantArt

  1. Go to https://www.deviantart.com/sandara/art/Hydra-2-828769182 (This is one image which is permited to Download by the author, but there are many more)
  2. Be sure to use the "Eclipse" layout (Choosing it using the switch in the top bar labeled as "Eclipse")
  3. Look below the image, the icons in the left, the third one (an arrow pointing down, if you hover above it, said "Download"). This is the Download Button.
  4. If you click the Download Button, it downloads the image normally, but:
     a) If you use DownThemAll (Mouse over Download button, secondary button menu -> "DownThemAll" -> "Save Link with OneClick!") it tries to download the image to the default download folder but fails.
     b) If you use "Save In..." (Mouse over Download button, secondary button menu -> "Save In..." -> ". (1)") it tries to download the image to the default download folder but fails, showing a "SERVER_BAD_CONTENT" message popup and a save dialog, but if you try to save with that fails again and nothing more.

• For Twitter

  1. Go to https://twitter.com/messages to look your Direct Messages (DMs) (You can also go to Twitter and click in the "Envelope" icon)
  2. Select a to send DMs to a contact, better an existing one with a thread with images sent via Twitter. If not create one
  3. Choose an image sent via Twitter in the thread. It can be sent by the owner or the contact.
  4. OPTIONAL: You can click above it to open it in big size or not.
  5. a) If you use DownThemAll (Mouse over the image, secondary button menu -> "DownThemAll" -> "Save image with OneClick!") it does nothing.
     b) If you use "Save In..." (Mouse over the image, secondary button menu -> "Save In..." -> ". (1)") it tries to download the image to the default download folder but fails, showing a "SERVER_BAD_CONTENT" message popup and a save dialog, but if you try to save with that fails again and nothing more.
     c) If you use "Save Image in Folder" (Mouse over the image, secondary button menu -> "Save Image in Folder" -> "Photos" or any other) it does nothing.

• For Tweetdeck

  1. Go to https://tweetdeck.twitter.com/
  2. Search the column "Messages @YourUser" (If you don't have it, add the column with the "+" "Add Column" option in Tweetdeck)
  3. Pick a contact in the message list. One with images sent via Tweetdeck/Twitter. It can be sent by the owner or the contact.
  4. OPTIONAL: You can click above it to open it in big size or not.
  5. a) If you use DownThemAll (Mouse over the image, secondary button menu -> "DownThemAll" -> "Save image with OneClick!") it does nothing.
     b) If you use "Save In..." (Mouse over the image, secondary button menu -> "Save In..." -> ". (1)") it tries to download the image to the default download folder but fails, showing a "SERVER_BAD_CONTENT" message popup and a save dialog, but if you try to save with that fails again and nothing more.
     c) If you use "Save Image in Folder" (Mouse over the image, secondary button menu -> "Save Image in Folder" -> "Photos" or any other) it does nothing.

• In all the cases (With DeviantArt, Twitter or Tweetdeck), the stock Firefox menu "Save Image/Link" works without problem.

Expected behavior (In all cases):

• If you use DownThemAll (Mouse over the image/link, secondary button menu -> "DownThemAll" -> "Save image/link with OneClick!") it downloads the image in the default download folder.
• If you use "Save In..." (Mouse over the image/link, secondary button menu -> "Save In..." -> ". (1)") it downloads the image in the default download folder.
• If you use "Save Image in Folder" (Mouse over the image, secondary button menu -> "Save Image in Folder" -> "Photos" or any other) it does download the image in a folder with the name selected in the "Save image in Folder" list, inside the default download folder.

All these expected behaviour happens with images in public tweets in my Twitter timeline, for example.

Additional Context:

• Maybe is related with a change of the filename during the dialog with the server. As i explain above in the first comment, the example link from DeviantArt, the filename for the image is first ddpfee6-011a5b5f-386b-40a8-849e-7f28a9dfb5dd.jpg, and it changes to hydra_2_by_sandara_ddpfee6.jpg with it's the desired filename. You can see it changing when DownThemAll! is trying to download it (It's fist the first (weird) one, it changes to the second (desired) one, and it revert back to the first (weird) one when it fails. When you try to download the image with clicking the link without any extension, it download with the second (desired) filename. Twitter seems to do something similar because sometimes appears the jpg, sometimes don't, sometimes with a large, medium etc. in the filename...
• As i explain above about saving the image directly in DeviantArt and not using the download button, saving the image doesn't maintain always the "desired" filename (with the author and name of the image) and download the image with the "weird" filename (This also happens with the stock Save image in Firefox, without extensions). Contacted the DeviantArt support, the right way is using the Download Button, which always downloads with the "desired" filename (but not with the Extensions as i am explaining because they fail to download).

I hope i was enough detailed. I tested all these with a clean profile as it was requested, if it's anything more i can do...
Thanks!

Thanks. I'll need to take a closer look.

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is P3 (Backlog,) indicating it has been triaged, the bug's Severity is being updated to S3 (normal.)

Thanks for the STR in comment 4. I followed the DeviantArt STR and managed to reproduce the issue.

DeviantArt assigns SameSite=Lax to several authentication cookies, which meant that the cookies are not included in requests from third parties.
Until very recently, requests from extensions were considered third-party, which meant that extension-generated downloads weren't considered first-party.

Since the initial report, some work has been done on handling third-partiness, cookies and the download APIs, which all contribute to this bug being fixed:

• Before any fix: SameSite=Lax and SameSite=Strict cookies not included, only SameSite=None.
• SameSite=Lax cookies included in requests: bug 1640405 (this specific fix fixes the DeviantArt bug)
• SameSite=Strict cookies included in requests as well: since bug 1579911 (relying on 1629436), provided that the extension has host permissions for the given URL.

I think that it may make sense to not require host permissions for SameSite=Strict cookies. That would be bug 1583715.
I'll add some unit tests to verify that the feature works as intended.

When running mozregression to find relevant bugs, I noticed that bug 1640405 allows downloads to see SameSite=Lax cookies, but not SameSite=Strict.

In bug 1579911, the loadingPrincipal for downloads changed from the system to an extension principal. As a result, SameSite=Strict cookies are included. This is surprising, is it intentional that requests with the system principal as the loadingPrincipal aren't including SameSite=Strict cookies?

(In reply to Rob Wu [:robwu] from comment #8)

When running mozregression to find relevant bugs, I noticed that bug 1640405 allows downloads to see SameSite=Lax cookies, but not SameSite=Strict.

Bug 1640405 is about treating "save-as" requests as safe-top-level navigation. In order to send strict cookies, we need to have a same-site request.

In bug 1579911, the loadingPrincipal for downloads changed from the system to an extension principal. As a result, SameSite=Strict cookies are included. This is surprising, is it intentional that requests with the system principal as the loadingPrincipal aren't including SameSite=Strict cookies?

I haven't tested this issue yet, but:

1. We use the triggering principal from the loadInfo request.
2. We do consider the system principal as first-party: https://searchfox.org/mozilla-central/rev/cf561cece0ca9aeaf0202e68699836d957d0c670/caps/BasePrincipal.cpp#559,587

Are you able to point where the same-site check fails when dealing with the downloading from the extension?

With Firefox 79, the issue with the DeviantArt site is gone. Now All extensions downloads the files normally.

But with Twitter and Tweetdeck DMs, the issue remains. Exactly the same behaviour as pointed above.

I've investigated comment 10 and found that it is not an issue with the downloads API itself.

(In reply to thywalls from comment #4)

• For Twitter
  1. Go to https://twitter.com/messages to look your Direct Messages (DMs) (You can also go to Twitter and click in the "Envelope" icon)
  2. Select a to send DMs to a contact, better an existing one with a thread with images sent via Twitter. If not create one
  3. Choose an image sent via Twitter in the thread. It can be sent by the owner or the contact.
  4. OPTIONAL: You can click above it to open it in big size or not.
  5. a) If you use DownThemAll (Mouse over the image, secondary button menu -> "DownThemAll" -> "Save image with OneClick!") it does nothing.

Reported as bug 1659155 and https://github.com/downthemall/downthemall/issues/273

b) If you use "Save In..." (Mouse over the image, secondary button menu -> "Save In..." -> ". (1)") it tries to download the image to the default download folder but fails, showing a "SERVER_BAD_CONTENT" message popup and a save dialog, but if you try to save with that fails again and nothing more.
c) If you use "Save Image in Folder" (Mouse over the image, secondary button menu -> "Save Image in Folder" -> "Photos" or any other) it does nothing.

I initially got 404 not found too when I tested the browser's built-in "Save Image As" option (on Twitter). But after downloading it once with fetch, I got succesful responses afterwards. If you are able to reproduce the issue consistently, please file a new bug.

Side note: For both the "Save In..." and "Save Image in Folder" add-ons, the implementations try to download the image from the server. If the image URL is valid only once, then the download can fail.
When I test https://jsfiddle.net/r94upv0g/ with these two add-ons, the download fails with "SERVER_FAILED" because the server replies with HTTP 415. This is because the chosen endpoint (https://httpbingo.org/image) serves a response based on the Accept HTTP header, and neither of those add-ons set the Accept header to an image type.

I'm surprised that the bug was still open, but apparently I did write the patch and pushed them to try, but forgot to actually upload the patches to Phabricator... I'll do that today.

Green try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=759517b0ec3f942eb7f752da06001d08bd02f7f5

This does not only provide coverage for cookie behavior in the
downloads.download API, but also the "mime" property of downloads,
which had zero test coverage.

https://hg.mozilla.org/mozilla-central/rev/7d672d57ea95

https://hg.mozilla.org/releases/mozilla-release/rev/222089930d48