User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0

Steps to reproduce:

1. Run a page that will generate audio periodically (e.g., a timer app, a system monitoring app) but where NO USER ACTION is desirable

2. Kill firefox (e.g., with SIGKILL)

3. Rerun to have firefox "restore" the session

Actual results:

4. Audio is blocked

5. After a day, the user noticed something is wrong and opens the developer console

6. Audio from the past 24 hours blast out from the speaker, potentially causing deafness

Expected results:

There need to be some way for pages that do not need user action to generate audio

When audio is unblocked it must not unblock all the pending audio all at once. The accumulated audio will be too loud to be safely played back.

This is worse than first suspected. In fact, a user action will not correctly unblock audio. So now you are breaking audio in perfectly correctly coded pages.

Steps 2 and 3 are not needed. Just loading a page that uses an audio interface will randomly break the page.

A page that is "perfectly correctly coded" will use the Promise returned by play() to determine whether or not audio playback is allowed, or the state of the AudioContext for the Web Audio API.

A user that wishes to not need a user action on the page to generate audio can add the domain to the list of domain that can play audio without user interaction, in the preferences, in the section "Privacy and security", "Block websites from automatically playing sound". This can also be disabled altogether.

Please give us a test case for comment 1. As far as we know, this works as intended, but it's hard to know for sure without a test case.

Several findings:

1. If the page has several audio contexts, unblocking one doesn’t unblock others. From the user’s POV this is just one confusing disaster.

2. From a keyboard user’s POV it’s not obvious what “user interaction” means. Again, just one confusing disaster.

Also, I need to reiterate that when audio is unblocked everything comes out right at once (e.g., periodic barely audio clicks can come out as a deafening shriek). This is an actual safety hazard.

Because safety is involved the only sensible way for a user to make exceptions is really to allow audio from all sites.

(In reply to Ambrose Li from comment #4)

1. If the page has several audio contexts, unblocking one doesn’t unblock others.

Also, I need to reiterate that when audio is unblocked everything comes out right at once (e.g., periodic barely audio clicks can come out as a deafening shriek).

Ambrose, is the periodic audio generated by several scheduled nodes in a single AudioContext?

Alastor, was this scenario considered in the decision to delay audio instead of muting audio?

I checked my code. I’m only using two audio contexts (one normal and one offline). I’ll have to trace my code more carefully to figure out what’s going on.

Re the earlier comment that correctly written code would check the returned promise. I seriously have to ask how you’re able to assume people even know this needs to be done. Most designers I bet are going to just copy and paste sample code. I copeid mine from the O’Reilly book and there’s no checks there. This audio blocking is going to break everything and no one is going to know what happened.

Can you just at least fix the UI? Pop up a yellow window with Continue blocking and Unblock buttons. The users will (1) know audio is being blocked (versus the current situation where a user could go on wondering why something isn’t working until a day later) and (2) they’ll have a sure way to unblock it (whereas currently I have no idea if my “user interaction” is enough to unblock). There shoud also be a preference to disable blocking for all sites.

Also, based on the discussion so far you are probably unaware that if a tab gets swapped out Firefox will “forget” audio had been unblocked. So any unblocked tab can unexpectedly get re-blocked for no reason and a page that gets re-blocked can take forever to get unblocked again.

(In reply to Ambrose Li from comment #4)

1. If the page has several audio contexts, unblocking one doesn’t unblock others. From the user’s POV this is just one confusing disaster.

Take media element for an example, unblocking one media element doesn't mean that we have to unblock other media element as well, user can play the media element which they are interest in, not force all of them to start at the same time.

So does AudioContext, every blocked AudioContext are unrelated with each other, the sites should handle it well by themselves if they decide to use multiple contexts.

(In reply to Ambrose Li from comment #6)

Re the earlier comment that correctly written code would check the returned promise. I seriously have to ask how you’re able to assume people even know this needs to be done. Most designers I bet are going to just copy and paste sample code. I copeid mine from the O’Reilly book and there’s no checks there. This audio blocking is going to break everything and no one is going to know what happened.

Allmost all browser venders have annouced their own post about how to write the proper code in the situation user block autoplay.

https://hacks.mozilla.org/2019/02/firefox-66-to-block-automatically-playing-audible-video-and-audio/
https://developers.google.com/web/updates/2017/09/autoplay-policy-changes
https://webkit.org/blog/7734/auto-play-policy-changes-for-macos/
https://docs.microsoft.com/en-us/microsoft-edge/dev-guide/browser-features/autoplay-policies

Can you just at least fix the UI? Pop up a yellow window with Continue blocking and Unblock buttons. The users will (1) know audio is being blocked (versus the current situation where a user could go on wondering why something isn’t working until a day later) and (2) they’ll have a sure way to unblock it (whereas currently I have no idea if my “user interaction” is enough to unblock). There shoud also be a preference to disable blocking for all sites.

We've tried to use door-hanger (which is like showing a obivous pop-out window to tell users that the media is blocked) in 63 via a experimental way, but the result is not good enough to keep it as a default.

The current UX design is that, it would show a blocking icon near the URL bar and the site information center, you can add or remove the site into user's own preference list to control whether you would like to allow/deny this site from autoplay.

(In reply to Karl Tomlinson (:karlt) from comment #5)

Alastor, was this scenario considered in the decision to delay audio instead of muting audio?

What we do is not muting the audio, we prevent the entire AudioContext from running. I think the issue here is, how AudioContext handle its delayed audio input during it's suspended? It sounds like this issue would also happen without blocking autoplay if the site is continually produce sound to an AudioContext which is explictly suspended by user.

How do web audio handle this kind of situation?

(In reply to Ambrose Li from comment #7)

Also, based on the discussion so far you are probably unaware that if a tab gets swapped out Firefox will “forget” audio had been unblocked. So any unblocked tab can unexpectedly get re-blocked for no reason and a page that gets re-blocked can take forever to get unblocked again.

I sounds like an another different bug, could you file a bug for that?
Thank you.

(In reply to Alastor Wu [:alwu] from comment #9)

What we do is not muting the audio, we prevent the entire AudioContext from running. I think the issue here is, how AudioContext handle its delayed audio input during it's suspended? It sounds like this issue would also happen without blocking autoplay if the site is continually produce sound to an AudioContext which is explictly suspended by user.

How do web audio handle this kind of situation?

If a sound is scheduled to play multiple times at the same time (by using multiple AudioBufferSourceNodes, for example), then Web Audio will play the sound multiple times at the same time. The effect will be that the sound is heard at one time, but the volume is amplified according to the number of times the sound is scheduled.

I assume the effect is similar if play() is called on multiple audio elements, each having the same sound. However, audio elements are not scheduled so precisely, and so I guess they may be heard as distinct sounds and the amplification effect will be somewhat less.

The suspension and resumption of the AudioContext only comes into play here if the scheduled times for the sounds are based on AudioContext.currentTime.

If the AudioContext were explicitly suspended by the client and many sounds were scheduled at currentTime, then they would all begin playing when the context would be explicitly resumed. This behavior would be expected by the client.

The issue here AIUI is that the client is using a timer or other mechanism in attempt to schedule sounds at different times. The unexpected consequence of the delays introduced by the audio-blocking implementation is that all of the sounds are played at once, with more-or-less unbounded volume.

I'm going to mark this P2 for now.

Sorry - while in unconfirmed state, priority should probably stay blank.

(In reply to Karl Tomlinson (:karlt) from comment #11)

The suspension and resumption of the AudioContext only comes into play here if the scheduled times for the sounds are based on AudioContext.currentTime.

I’m at a loss for words. Anyone using the Web Audio API would schedule their sounds based on AudioConext.currentTime (calibrated against the system clock if needed, of course); otherwise your sounds aren’t synchronized properly.

Using the regular Javascript timer used to be possible, but no longer because the granularity has become too low due to Spectre/Meltdown mitigation.

I wrote up a simple sample page for this at https://jsfiddle.net/pehrsons/b75jgze3/

It plays a stapler-sound every second. Make sure jsfiddle's autoplay is blocked and wait a minute before interacting to hear the symptoms reported here.

This does seem breaking to existing sites to me. New sites could of course be blocking-agnostic and handle this, but if they don't it could get very loud on first interaction (a game?).

Alastor, could you drive this forward? Whatever the correct resolution could be. For now, marking this confirmed, P2.

(In reply to Karl Tomlinson (:karlt) from comment #11)

If a sound is scheduled to play multiple times at the same time (by using multiple AudioBufferSourceNodes, for example), then Web Audio will play the sound multiple times at the same time. The effect will be that the sound is heard at one time, but the volume is amplified according to the number of times the sound is scheduled.

Is it the spec defined behavior? Because I think the problem here is not related with whether we block web audio or not, the issue would happen on any suspended AudioContext. The issue is why we amplify the sound in this situation.

In this sample [1], the AudioContext was suspended by user explicitly, and it would be resumed when user clicks on the page. Then, bomb, the sound blasts if you wait for long enough.

[1] https://jsfiddle.net/ubkdm4q7/4/

(In reply to Alastor Wu [:alwu] from comment #16)

(In reply to Karl Tomlinson (:karlt) from comment #11)

If a sound is scheduled to play multiple times at the same time (by using multiple AudioBufferSourceNodes, for example), then Web Audio will play the sound multiple times at the same time. The effect will be that the sound is heard at one time, but the volume is amplified according to the number of times the sound is scheduled.

Is it the spec defined behavior?

Yes, it is specified that multiple sounds played together through Web Audio will add together.

Because I think the problem here is not related with whether we block web audio or not, the issue would happen on any suspended AudioContext.

The author would not usually suspend the AudioContext and then schedule many sounds together, because authors know the effect this will have.

Even before blocking was implemented, however, there was the potential for malicious authors to play very loud sounds. Perhaps we should apply soft clipping to mitigate that.

The issue is why we amplify the sound in this situation.

This bug report was filed because the blocking implementation changes behavior to produce very loud sounds in situations where the author did not intend this.

Andreas, was there a product manager involved in the design of the audio blocking implementation?

The question is were these consequences considered in making the decision to play blocked audio after arbitrarily long delays?

(In reply to Alastor Wu [:alwu] from comment #16)

(In reply to Karl Tomlinson (:karlt) from comment #11)

If a sound is scheduled to play multiple times at the same time [...]. The effect will be that the sound is heard at one time, but the volume is amplified according to the number of times the sound is scheduled.

Is it the spec defined behavior? Because I think the problem here is not related with whether we block web audio or not, the issue would happen on any suspended AudioContext. The issue is why we amplify the sound in this situation.

This is how audio works. If this is not spec-defined behaviour then the spec is wrong and no one will use it because it will be useless.

(In reply to Karl Tomlinson (:karlt) from comment #17)

Even before blocking was implemented, however, there was the potential for malicious authors to play very loud sounds. Perhaps we should apply soft clipping to mitigate that.

Yep, as the issuse has existed before we started blocking autoplay, it's just not easy to be noticed by common ways web developers use, I think that is the best way to handle this issue. Maybe we can restrict the maximum ampified thershold for playing this kind of audio?

(In reply to Alastor Wu [:alwu] from comment #19)

(In reply to Karl Tomlinson (:karlt) from comment #17)

Even before blocking was implemented, however, there was the potential for malicious authors to play very loud sounds. Perhaps we should apply soft clipping to mitigate that.

Yep, as the issuse has existed before we started blocking autoplay, it's just not easy to be noticed by common ways web developers use, I think that is the best way to handle this issue. Maybe we can restrict the maximum ampified thershold for playing this kind of audio?

One question is, who gets to define “loud”? Audio behaves differently depending on whether you are using headphones, normal computer speakers, or speakers through an amplifier (e.g., for art projects, professional audio production, etc.).

I suppose you could argue if the audio output goes through an amp then the audio source doesn’t need to be “loud”, but if it’s too soft then there’d be too much noise when amplified so it’s probably not that good either.

(In reply to Alastor Wu [:alwu] from comment #19)

(In reply to Karl Tomlinson (:karlt) from comment #17)

Even before blocking was implemented, however, there was the potential for malicious authors to play very loud sounds. Perhaps we should apply soft clipping to mitigate that.

Yep, as the issuse has existed before we started blocking autoplay, it's just not easy to be noticed by common ways web developers use, I think that is the best way to handle this issue. Maybe we can restrict the maximum ampified thershold for playing this kind of audio?

Sorry, I had assumed that it was possible for authors to override system volume by playing a signal with greater than unit amplitude. However, in my experimentation, with Pulse and a Win 7 / Realtek system, it seems the system is already limiting the volume (clipping with Pulse, and compression on the Win 7 system).

Perhaps there are some other systems that don't limit at user-indicated levels. I don't know, but even if such systems exist, any mitigation we implemented could only apply at a maximum volume. The issue reported here would still cause sound to be louder than the author intended.

This bug is questioning the usefulness of playing sounds intended for long before the audio is unblocked.

I don't know what to do here. We already provide a way to not block, per origin or globally, if particular people are concerned with this.

More over, AudioContext.state has been implemented five years ago in Firefox (probably before in Chrome and others), and it should be pretty clear by now that scheduling a lot of sound when the context is suspended at the AudioContext.currentTime is going to play something loudly when it's restarted, regardless of the way the context was suspended.

Inserting a look-ahead limiter on the audio path isn't a good solution because that adds latency, and a limiter that is not look-ahead is bound to not catch the problem. Also, any dynamic shaping is potentially deteriorating the audio for all apps. Because some of them are not observing AudioContext.state doesn't seem like a compelling option. Doing hard clipping is another option, but clearly using the extra dynamic range at the output is useful and being used in a lot of programs, and it generally doesn't clip hard, because we're using f32 on desktop platforms.

Ramping the volume up on resume doesn't work because we don't know the length of what's been scheduled, and as noted earlier, we don't even know the output device, we're just output audio as floats.

Do other implementations do something differently?

Closing this, but happy to hear about possible solutions/mitigations.