The basic idea here is to allow someone to write a kind of add-on, called a native.js add-on, that has the following properties: 1. loads using the bootstrap.js scheme 2. has access to XPCOM through Components 3. can modify XUL via JS, but would not be able to include XUL files or overlays 4. extends the WebExtension APIs with new APIs 5. use a simple JSON manifest 6. no chrome.manifest or install.rdf files These add-ons would not have any UI of their own. WebExtensions could mark themselves as dependent on the APIs exposed by these add-ons. When installing such a WebExtension, the native.js add-on would be downloaded and installed automatically from AMO or something. I think we need to implement OOP WebExtensions before we do this. Otherwise I worry that the system for extending WebExtension APIs wouldn't work when we finally implement OOP WebExtensions.

In the triage meeting we decided to file a "preliminary native.js bug". But I realized that this bug is already for a prototype. I'll file a different bug for full native.js support. This bug will block 1.0 and the other one won't.

(In reply to Bill McCloskey (:billm) from comment #0) > The basic idea here is to allow someone to write a kind of add-on, called a > native.js add-on, that has the following properties: > > 1. loads using the bootstrap.js scheme > 2. has access to XPCOM through Components > 3. can modify XUL via JS, but would not be able to include XUL files or > overlays > 4. extends the WebExtension APIs with new APIs > 5. use a simple JSON manifest > 6. no chrome.manifest or install.rdf files Why is the plan now to strip the add-on sdk of access to XUL and XPCOM and yet provide it to webextensions? With the sdk the `require('chrome')` access could be compartmentalized in to a module which could be shared on npm and tested with unit tests that ran on travis-ci, and eventually could easily land in Firefox. What is the story for native.js? is there no module sharing? nor an ability to reusable code without copying and pasting? and will there be an easy way to write tests for these features like the sdk provided? is add-on testing still something Mozilla wants to support? if so then how will writing tests for this look?

Wouldn't it make sense to have native.js modules/addons/whatever work for both webextensions and sdk addons. Or make the webextension APIs available to SDK addons so that they transitively have access to native.js?

native.js code will be packaged as a separate extension. These extensions will be updated independently of the extensions that depend on them. One of the problems with require('chrome') is that, even though the modules are re-usable, each extension includes its own copy of the code. We definitely intend for successful native.js APIs to be integrated into Firefox. native.js extensions will be incubators for new APIs. As such, they'll probably change a lot and Mozilla keep a close eye on their development. I haven't thought much about tests. We have a testing framework (the browser.test API) that we use internally, but it's not usable by add-on developers. That should probably be fixed. Thank you for filing the bug. I don't think anyone is opposed to giving the SDK the same access to native.js as WebExtensions. Right now our first priority is for the SDK to be fully e10s-compatible, though, and that will probably take time.

(In reply to Bill McCloskey (:billm) from comment #4) > native.js code will be packaged as a separate extension. These extensions > will be updated independently of the extensions that depend on them. This is a little concerning, but not terrible I suppose, what happens if there is an api change? Anyhow, I don't see why we need a new extension type when we could easily create a "chrome" permission for Jetpacks that effectively makes the add-on a "native.js extension", and have the same results as a native.js extension thing, and on top of this the apis would automatically work with Jetpacks too. > One of > the problems with require('chrome') is that, even though the modules are > re-usable, each extension includes its own copy of the code. Well that is fixable I think, if we make using `require('chrome')` a permission and let these add-ons extend both the sdk and google chrome apis. Instead it sounds like we are inventing yet another extension type, which can only extend google chrome apis easily, and which cannot use code from npm easily. Also this seems like it will be complicated and use more resources. > We definitely intend for successful native.js APIs to be integrated into > Firefox. native.js extensions will be incubators for new APIs. As such, > they'll probably change a lot and Mozilla keep a close eye on their > development. > This is what the sdk was planning to do so it's not a benefit afaict, this goal will actually be harder to accomplish than with the sdk approach for the reasons I mention above. > I don't think anyone is opposed to giving the SDK the same access to > native.js as WebExtensions. Right now our first priority is for the SDK to > be fully e10s-compatible, though, and that will probably take time. I have looked into how much work this will be, this should be pretty easy to do in a couple of weeks for one person.

(In reply to Erik Vold from comment #5) > I have looked into how much work this will be, this should be pretty easy to > do in a couple of weeks for one person. I have no doubt if that person would be you that could work, but right now I don't see anyone from the old SDK team working on it, and for someone else getting started on the SDK, it is not a fast process (because of the very same architectural decisions that slowed down the project for example).

(In reply to Erik Vold from comment #5) > (In reply to Bill McCloskey (:billm) from comment #4) > > native.js code will be packaged as a separate extension. These extensions > > will be updated independently of the extensions that depend on them. > > This is a little concerning, but not terrible I suppose, what happens if > there is an api change? Bill? > Anyhow, I don't see why we need a new extension type when we could easily > create a "chrome" permission for Jetpacks that effectively makes the add-on > a "native.js extension", and have the same results as a native.js extension > thing, and on top of this the apis would automatically work with Jetpacks > too. > > > One of > > the problems with require('chrome') is that, even though the modules are > > re-usable, each extension includes its own copy of the code. > > Well that is fixable I think, if we make using `require('chrome')` a > permission and let these add-ons extend both the sdk and google chrome apis. > Instead it sounds like we are inventing yet another extension type, which > can only extend google chrome apis easily, and which cannot use code from > npm easily. Also this seems like it will be complicated and use more > resources. > Am I wrong Bill? if I am right then native.js is useless and a huge waste of time. > > > We definitely intend for successful native.js APIs to be integrated into > > Firefox. native.js extensions will be incubators for new APIs. As such, > > they'll probably change a lot and Mozilla keep a close eye on their > > development. > > > > This is what the sdk was planning to do so it's not a benefit afaict, this > goal will actually be harder to accomplish than with the sdk approach for > the reasons I mention above. Am I wrong here Bill? how will developing native.js things without shared code from a package manager like npm not make life harder for extensions developers unnecessarily? ni'ing Andy since he appears to be the PM on this, and these questions really need answering before e10s is forced on add-on devs. It seems awfully foolish to me for Mozilla to tell add-on devs to first support e10s natively without the needed web extensions apis existing or some way to extended these apis (because they aren't ready), then force add-ons devs to re-write everthing to support web extensions.. if this is your plan then go ahead and remove the ni..

Can someone please mark this P1 by the way? some resolution here is absolutely necessary in order for the following statement from Kev Needham to be true: "We want to reiterate our commitment to our add-on development community, and will work with you in porting extensions, designing new APIs, and creating innovative new add-ons that make Firefox great." From https://blog.mozilla.org/addons/2015/08/21/the-future-of-developing-firefox-add-ons/

Andy how will this native.js plan work on the AMO side, since it is very likely that any one extension will depend on many, like 10+, little native.js extensions?

One more thing... Let's say Mozilla has already spent the resources required to make yet another add-on type, native.js. What is going to prevent someone from building their entire add-on as a native.js thing in order to avoid using web extension apis?

(In reply to Erik Vold from comment #9) > Can someone please mark this P1 by the way? some resolution here is > absolutely necessary in order for the following statement from Kev Needham > to be true: > > "We want to reiterate our commitment to our add-on development community, > and will work with you in porting extensions, designing new APIs, and > creating innovative new add-ons that make Firefox great." I disagree. Working with authors, designing new APIs, and creating innovative new add-ons may be helped by native.js, but the volume of interesting Chrome add-ons are a strong argument against it being necessary. (In reply to Erik Vold from comment #11) > Let's say Mozilla has already spent the resources required to make yet > another add-on type, native.js. What is going to prevent someone from > building their entire add-on as a native.js thing in order to avoid using > web extension apis? This bug is hardly the place for this discussion. Please take it to one of the communication channels listed at https://wiki.mozilla.org/WebExtensions#Communication_and_meetings. Thanks, Blake.

(In reply to Erik Vold from comment #11) > Let's say Mozilla has already spent the resources required to make yet > another add-on type, native.js. What is going to prevent someone from > building their entire add-on as a native.js thing in order to avoid using > web extension apis? I have contributed to a few addons and from the addon dev perspective the migration plan seems to be more like using bootstrap/sdk extensions for now and import and use web extension APIs into the old addon where possible and not the other way around. Switching to native.js only makes sense when there are few pieces remaining that cannot be done with web extensions. Anyway, in my opinion its mostly a matter of carrot and stick policy. With the right review guidelines for native.js addons and some "tainting" for addons relying on them that could result in clean, minimal APIs. for example: - addons with native.js dependencies must feature-detect their presence -> mozilla can nuke the latter from orbit without hitting the former - mandatory code documentation for the exposed API to allow others to re-implement or re-use it - minimal, clean APIs (callbacks/promises, only structured clones, ...) - (optional) tests to make checking whether a re-implementation is compatible / whether it's compatible with new FF versions I think especially the first point will be of use, since it will bring them closer to experimental web APIs that people already use today via feature detection and it would turn native.js dependencies into optional dependencies, which might make addon management a little bit easier. (In reply to Blake Winton (:bwinton) (:☕️) from comment #12) > I disagree. Working with authors, designing new APIs, and creating > innovative new add-ons may be helped by native.js, but the volume of > interesting Chrome add-ons are a strong argument against it being necessary. My impression is that some interesting chrome addons came into existence be chrome looking at "what did some firefox addons invent, how can we support this?". And some more outside-the-box things simply cannot be done in chrome.

At Mozlando a bunch of mozillians got together and hashed out a plan for how native.js would work (and maybe a better name). We'll be getting a Wiki page up on the proposal soon and email dev-addons (any maybe a few others) to get feedback.

Is it done? Can you post a link to that page?

(In reply to Andy McKay [:andym] from comment #14) > At Mozlando a bunch of mozillians got together and hashed out a plan for how > native.js would work (and maybe a better name). We'll be getting a Wiki page > up on the proposal soon and email dev-addons (any maybe a few others) to get > feedback. Did you write it down? Can you post a link to that page?

Kris is planning on working on this for bug 1263011, with a slightly different focus. I think we can probably close this bug in favour of that one, however the scope of that bug is slightly more smaller than this.

We've got a working Experiments implementation and are working on that this quarter. It seems to work well. http://wiki.mozilla.org/WebExtensions/Experiments http://github.com/web-ext-experiments/ I realise its not *exactly* the same as this, but pretty close. Enough to close this?

No, Andy McKay, 'Experiments' is *nothing* like this. Please clarify "pretty close"; I've *no* idea what you're thinking here. native.js [1] was to allow an extension continued access to the chrome process by separating it into two components, a headless chrome-process component and an extension-process component with the UI, connected by cross-process messaging like WebExtension background pages and content scripts. Like a background page, the chrome-process component could communicate with any WebExtension. This allowed 2+ WebExtensions to use the same chrome-process backend. In time, Mozilla would adopt popular back-ends, rationalize them, and formalize them as well-known messaging addresses or as `chrome.*` methods. native.js is a brilliant proposal. (1) It provides XUL- and XPCOM-dependent extensions a migration path to WebExtensions, and provides a way to expose existing .jsm functionality to WEs. (Like thumbnails?) Separating the code by compartment (2) allows reviewers to apply stricter standards to "riskier" code. That (3) discourages developers from running chrome-process code ("ack! review so slow!") but (4) doesn't prevent them from doing it when they must. That (5) preserves the versatility of Firefox as an extension platform while still (6) improving oversight of the riskier elements of more powerful extensions. It (7) permits developers to freely devise and experiment with new ad hoc interfaces to chrome-process functionality; these are the (8) feedstock for new canonical interfaces. If chrome-extension messaging is modeled on cross-extension messaging then (9) there is a simple way to monitor interface usage. (10) It neatly extends Chrome's two-compartment "separation of powers" extensions model to three compartments using only conventions and interfaces that are already familiar. Nothing novel or ground-breaking is required. 'Experiments' [2] offers none of those advantages. It is merely an extension-installing extension, with telemetry. As an afterthought you propose an API-extending extension as one of those subsidiary extensions. However, its only point of contact with native.js is as a test jig for vetting proposed interfaces before adopting them into the API. It is useless to extension developers. A developer who needs chrome access will do what Experiments developers have done - write a bootstrapped add-on and ignore the WebExtensions API altogether. Truth be told, I don't really see any utility in 'Experiments' at all. I mean, why wouldn't you just distribute your experiments like any other add-on? They don't seem much different than add-ons that are already available there. [1] https://discourse.mozilla-community.org/t/proposal-native-js-to-embrace-extend-the-webextensions-api/3457 [2] http://wiki.mozilla.org/WebExtensions/Experiments [3] https://testpilot.firefox.com/privacy

Fair enough. > Truth be told, I don't really see any utility in 'Experiments' at all. I > mean, why wouldn't you just distribute your experiments like any other > add-on? They don't seem much different than add-ons that are already > available there. With an Experiment you can write a WebExtensions API without having to build or commit to mozilla central. Experiments are going to be distributed out to dev edition and nightly by default. That means people can hack and ship out to those channels quickly without being tied to the release trains.

But as an extension developer who needs a "unique" API - one that likely no other extension will ever use, are Experiments going to help me? Can I maintain my personal API in order to keep the extension working? One example where this would be necessary is my JavaScript Deobfuscator extension. It needs some information from the JavaScript debugger APIs that Developer Tools normally don't collect. There is little doubt that Mozilla won't commit to adding and maintaining this functionality. I wouldn't mind doing so myself, as I've done in the past. Will I be able to create and maintain my own Experiment? Will I be able to distribute it to people who want to install JavaScript Deobfuscator without adding too many hurdles for them to jump over?

(In reply to Wladimir Palant from comment #21) > But as an extension developer who needs a "unique" API - one that likely no > other extension will ever use, are Experiments going to help me? Can I > maintain my personal API in order to keep the extension working? In their current incarnation, yes you would be to do so - but only on nightly or developer edition, because experiments won't work on release versions. There's lots of questions to answer before we feel this could roll out onto release. Amongst them is how we feel about developers being able to effectively completely circumvent the WebExtensions sandbox and do whatever they'd like. You'll see discussion about on dev-addons.

I have a child_process experiment here...hm nice to have it but as long it is not working on release channel its simple not useable for Addon developers. Now could be the idea here that people upload their experiments to Github for integration in Firefox. But who wants to spend time to develop an API and behind this you have to hope that somebody on Mozilla side find this useable and vote for merge to firefox? So experiments are a tool to develop APIs from Mozilla devs for Mozilla devs.

Why don't you just implement native.js as it is ? It's a perfect compromise that takes into account the need for Firefox to be freed from extension hooks without hindering the potential for innovation through extensions. It has been discussed at length and now you discard it without visible discussion ? Your project is quite different and sounds quite a lot more limiting without freeing Firefox that much more than native.js. I don't get it. If there's some place where this is discussed in detail, please point me to it, but a discussion has to occur.

This bug has not been closed. No-one has stepped up to implement native.js and we've done work on Experiments as a first step towards it. We are more than happy to have discussion about it on dev-addons and other mailing lists. https://wiki.mozilla.org/Add-ons#Getting_in_touch

Oh, that's so disingenuous. "No-one has stepped up"? - no one stepped up to implement Experiments, either. You waved your magic wand and assigned someone to do the heavy lifting. (Or did he do it pro bono?) You might as easily have conjured some of :billm's time to prototype native.js. Until three weeks ago :billm owned this but was waiting (comment 0) for extensions to move into their own process (bug 1190679). Three weeks ago he unassigned himself and you immediately arrived to hang crepe. No more of this "no-one has stepped up" nonsense. It's simply not true. The problem, I think, is that you do want a richer API but you don't want any extension code running in the chrome process, so you devised an extension ghetto where "experimental" extensions could run privileged code as experimental API methods in experimental Firefox. If Mozilla adopts the experimental API, then the privileged code can be removed and replaced with the new API calls and the "experimental" extension can finally be distributed at AMO. Of course, that means that extensions with chrome dependencies will ride the "release train" waiting for new APIs to land. Not good. The work you've done on Experiments is as easily the basis of a "pre-'Update'" [2] native.js prototype. In aswan's example [1], you'd start a message listener in the chrome process (api.js) instead of subclassing `ExtensionAPI`, then send messages from the background page instead of calling ad hoc `browser.*` methods. No "meta-API" or "dependency resolution system", so no "needs of the many" metric for deciding which ad hoc APIs to adopt, but Experiments has the same deficit. (Does it? The documentation is very sketchy.) The question is, will Mozilla permit any extension code to run in the chrome process? If not, then we're stuck with something like Experiments, and extremely long development times for extensions that touch the chrome. (Also, it means we've been snookered, which is upsetting.) So, will it? Also, can you link to those threads you mentioned above, Andy McKay? My search-fu is failing me. [1] https://github.com/aswan/webext-experiment-hello [2] https://discourse.mozilla-community.org/t/proposal-native-js-to-embrace-extend-the-webextensions-api/3457

(In reply to Nancy Grossman from comment #26) > The question is, will Mozilla permit any extension code to run in the chrome > process? If not, then we're stuck with something like Experiments, and > extremely long development times for extensions that touch the chrome. > (Also, it means we've been snookered, which is upsetting.) So, will it? I personally don't think it should. But that's my personal opinion and a discussion that explores the reasoning behind that would be good since what we allow add-ons to do affects so many people. I was referring to the thread: https://mail.mozilla.org/pipermail/dev-addons/2016-September/001898.html Which is a rather long thread that gets into Experiments, what should be allowed on release etc. A new "native.js is happening?" thread wouldn't hurt since lots of people are chatting about it.

> I personally don't think it should. I think it should as long as doing that is necessary to do things that firefox itself does. If the parent process is reduced to some thin broker and the whole browser, including privileged code, is split over a bunch of processes then we obviously wouldn't need addon code to run in the parent process anymore. We would need it to run in whichever process is responsible for the privileged part. Which process it runs in really is a distraction here. The point in equal API access. For a long time now the promise of addons has been "mozilla won't do it, but you can do it yourself". Many enhancements that people wanted were rejected with the reasoning that an addon would be a better place to do them. Those requests will always contain a long tail that will exceed mozilla's engineering capacity to accommodate. That's why it's important to let people take things into their own hands with minimal involvement from mozilla. I see two options here a) you only intend to accommodate the short end in the future and lock everyone else out. then please say so now. b) we get the same access as FF code itself has, one way or another

(In reply to Andy McKay [:andym] from comment #18) > We've got a working Experiments implementation and are working on that this > quarter. It seems to work well. > > http://wiki.mozilla.org/WebExtensions/Experiments > http://github.com/web-ext-experiments/ > > I realise its not *exactly* the same as this, but pretty close. Enough to > close this? No, of course not. You must surely be aware of this. It's not enough. Are you really so willing to throw away Firefox's key USP? Are you so detached from the user base not to recognise what it is that makes Firefox different (and thus successful)? Experiments is experimental. According to <https://wiki.mozilla.org/WebExtensions/Experiments>: "To repeat, this is the overall goal: be an area to allow developers to develop and experiment with APIs." This is quite clearly NOT a way to ship working extensions, which is the point of native.js. WebExtensions on their own will unnecessarily and unreasonably limit flexibility of extensions until you get on and implement native.js. What is the delay? To claim, as you did in https://bugzilla.mozilla.org/show_bug.cgi?id=1199718#c25, that no one "stepped up" to work on native.js is at best disingenuous, as Nancy Grossman has already pointed out. How many resources went into creating Experiments that could have been directed to native.js?

(In reply to The 8472 from comment #28) > Which process it runs in really is a distraction here. The point in equal > API access. If you take a look at the "Followup: modularity, WebExtensions, and going faster" currently ongoing on the firefox-dev list, you'll see that equal API access is seen as a bug by @mozilla.com contributors, rather than a key feature. This seems to me to be a fundamental disconnect between @mozilla.com coders and staff and the expectations of Firefox end users and addons devs. You further point out the disconnect here:- > For a long time now the promise of addons has been "mozilla won't do it, but > you can do it yourself". Many enhancements that people wanted were rejected > with the reasoning that an addon would be a better place to do them. > > Those requests will always contain a long tail that will exceed mozilla's > engineering capacity to accommodate. That's why it's important to let people > take things into their own hands with minimal involvement from mozilla. > > I see two options here > > a) you only intend to accommodate the short end in the future and lock > everyone else out. then please say so now. > b) we get the same access as FF code itself has, one way or another The Mozilla side of this arrangement is now lacking in substance. WebExtensions are a good idea in and of themselves but as we all know they cannot and do not replace the full functionality that the "but you can do it yourself" compact between users (and addon devs) and Mozilla provided, the very compact that was surely, more than any other thing, the key USP for Firefox: The ability to make supremely flexible addons. This is a feature that Chrome does not have and it is (or was) a key USP of Firefox over Chrome. Throwing this ability away to emulate Chrome offers no advantages to Firefox. "Security" is not an overwhelming advantage in this context. Experiments are of course not native.js. The chance to help develop an API that might make it into Firefox at some time in the future (maybe even the near future) is not a useful or realistic alternative to developing an actual, working, capable, flexible extension.

The main use case for native.js are: 1) Provide a transition period for extensions to migrate. For this embedded WebExtensions has been written: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Embedded_WebExtensions. Its a similar idea to native.js but inverted - legacy add-ons embed WebExtensions. 2) Allow extensions to run non-WebExtension code. Based on the policy in https://blog.mozilla.org/addons/2016/11/23/add-ons-in-2017, it will not be possible to run anything but WebExtension code in release past Firefox 57. For those reasons I'm won't fixing this bug.

Andy, could you please write the rationale for closing the bug here, that is, the actual rationale behind the said policy. There isn't a single bit of information on the issue to be found anywhere, which is very weird, because there has to be an enormous reason to justify scrapping of native.js (or similar approaches). The only thing I can think of is a very small security benefit it could provide. Addon review cost is the other issue I can think of, but that one is addressed by WebExtensions+native.js with "only when necessary" policy. Ability of addons to modify browser UI code is the defining feature of FireFox. Considering that Mozilla evidently lags far behind Chrome in manpower and resources, there's very little FF can do to compete if this feature is lost. So, again, this is a "life and death" kind of issue for FireFox and there must be a "life and death" reason for scrapping native.js. What is it?

Please see blog posts: https://blog.mozilla.org/addons/2015/08/21/the-future-of-developing-firefox-add-ons/ You will note in there that in 2015 it was said we'd deprecate the permissive add-on model. And: https://blog.mozilla.org/addons/2016/11/23/add-ons-in-2017/ Also the many mailing list threads at dev-addons mailing list.