When viewing any page, we need to keep a copy of the source around, instead of opening again, in the case of the file protocol, or downloading it again, in the case of the http and ftp protocols. The reason this must be done is simple. Save As, in other applications, means, save what I am currently viewing to a file that I will specify. In most cases, this is what Mozilla does with it's Save As. However, due to the fact that we are regeting the page, there stands a chance that it has changed, or been completly removed. This goes completly against the 'save what I am currently viewing to a file' meaning most users know Save As to mean. One of the worst consequenses of this is for web developers. Say you are designing a page, and have a copy open in mozilla. You make an unwanted change to the source file, and are unable to undo it. Or maybe you even accidently deleted the source file. No worries, you have a copy cached in Mozilla, right? Wrong. Also note that trying to save it, after the file has been deleted, will crash Mozilla if the file was accessed via ftp or file. I've opened bug 40792 on this problem. If the file was accessed via http and was deleted, Mozilla will quietly save the 404 page. This is bad. Very, very bad. Another problem is pages with dynamic content. You want to save the page you are viewing. Maybe it was loading 2 hours ago, but thats the page you want saved, not whatever is there now. Yet another problem is large pages for users on dialup connections. The Lynx browser makes you redownload a page when you to view as source. I found this infuriating when I was a dialup'er, trying to speed up my surfing by using Lynx. It made no sense to have to redownload what I seemingly was staring right at. For the composer, a bug was filed a month ago (Bug 37023), which is related to this. davidr8 reported that composer was forcing you to save before you could view source. beppe commented that it was more efficient to read from a file, and the bug was set to VERIFIED WONTFIX. This is exactly the same problem. Do not read the source from a file. Say I make a page in composer, save it, then edit that saved file in another editor. If I switch back to composer, I'm looking at the old version of the file. If I view source, I want, and expect, to see the old source. What if i needed to reference it. It's seemingly there, hell, I'm staring right at it! Saving the pre-rendered page in memory is a huge bandwidth savings if it was accessed over http/ftp, and is just plain the right thing to do.

Great bug! I think it's the same as bug 6119, though.

That is only regarding view source. I proposed that the system be used for Save As, and view source. I'll put a note on that one.

Somehow, I would think the cache should take care of this, given a "Once Per Session" recheck pref (in Advanced|Cache in the prefs). Adding "perf" keyword.

Looks similar to bug 6119. Reassigning to Bill law. Neeti

I suggest that bug 6119, bug 17889, bug 37023, and bug 40792 (and probably a few other bugs as well) all be made dependent on this bug; and that this bug be upped to major severity, since it involves loss of data. Otherwise those other bugs are all going to run around doing various inappropriate things with the cache, when commands operating on the current file (save, print, view source, etc) shouldn't be dependent on the cache at all (e.g. if the sum total of files in the current Web page is larger than the disk cache size). This bug really isn't much to do with XML. Mozilla should hold the source of *all* files currently being used, whether they be XML, PNG, CSS, or whatever.

Changing component from XML to XPApps (as I think was intended) and nominating for nsbeta3 since I think this is a major problem - we need to use the cache for these things (and get things out of the cache even when expired in certain circumstances).

Nav triage team: Already works the way 4.x works.

Removing nsbeta3- to trigger re-evaluation. PDT: This causes silent data loss! If you want to save the page currently being shown, but the page in question is no longer on the server, or is dynamically created, then Mozilla will silently lose the page, saving the 404 file not found page, or the different version in the case of dynamic content, without telling the user. This is also a problem with View Source, one that content developers find incredibly annoying with dynamic content. Say there is a bug on a dynamically generated page. The user right clicks on the page to view the source, but when the source comes up it is for a totally different page, since the server has generated a whole new document. There are many other examples of when this is critical (e.g. dialup connection is dropped before user saves page, view source on a POST operation such as after a money transaction over the net, saving NetCenter content that changes regularly, and so on and so forth). IE does this correctly, so marking 4xp. Moving up to 'major' severity, since this is a major loss of function. Is there a keyword for this kind of bug? Marking 'correctness' unless someone can think of a more appropriate keyword...

[To reverse the old maxim: it's not a feature, it's a bug! Resummarizing.]

nav triage team: nsbeta3-, while we believe this to be important, we are concerned that this would be too difficult to do safely, this late in the game.

Sanity check: this bug is nsbeta3-, but blocks bug 17889 which is nsbeta3+.

Removing nsbeta3- again for re-evaluation. We have a lot more information on this bug now. Bugs 17889 and 6119 are closely related, and likely have the same fix. 17889 is nsbeta3+, and I just nominated 6119 for + and dogfood. Unless somebody can come up with a super-workaround for both bugs, this bug should really be fixed. Ian Hickson has excellent comments in bug 6119 describing the problem, and I copied comments from a dup of 6119 which may help towards a fix.

adding cc

See also bug 39957... Okay, so I'm thinking about this now. I'm not sure what we currently do, but I would imagine that we keep a DOM tree for our current document if it's HTML or XML, and otherwise, we just display the document source. It would probably not be a Good Thing to keep both a DOM and the source of a large document around at once, but it might be the compromise we would have to make for beta3. In the long run, what might be possible is to have our object model code for XML/HTML store enough information to be reserialized exactly as it was in the source, such as whitespace, capitalization, comments, entity substitutions, etc. A lot of thought is being given to this stuff in the W3C's work on the infoset (http://www.w3.org/TR/xml-infoset) and this sort of stuff has been mulled over plenty in SGML's "groves" -- the issue of what data constitutes the "information represented by the document" and what is also in the document source but is considered extraneous... Anyway, the XML and HTML parsers could store this extra information in the object model (and hopefully by this time we'll have a unified object model behind all our DOMs), and thus avoid any duplication of information. That would be the intelligent solution to the view-source, save-as, and send-page bugs (see bug 6119). The printing and changing charset bugs don't need the exact source, so this isn't a direct solution to those, but the DOM does need to get held in memory for them... Now that I think about it, why should the printing and changing charset bugs even exist, if the DOM is kept in memory already (which it must, right?) Does the document actually get reloaded and reparsed before doing those things? That would be really messed up. I'm thinking that it might be two different underlying bugs: (1) we need to hold the source for the view-source/save-as type bug, and (2) we need to make print, change charset, etc. use the current DOM we have in memory. Does this sound like a reasonable analysis?

nav triage team: there are other ways to fix the specific user problems (like reading from the cache) that is much less of a hit then this suggested fix. nsbeta3- M Future to consider later.

Johng, that is incorrect. You can't read from the cache for a page which Mozilla has been told not to cache, because it's not in the cache. For example -- you want to save, or print, the order confirmation page for something you've bought over the Web, but the server (correctly) has told Mozilla not to cache that page for privacy reasons. So Mozilla goes to the cache, finds that the page isn't there, and then *reloads the page* ... so (unless the shop site is quite cleverly designed) you end up inadvertently doubling your order. Need I say, that is not desirable?

I'm not 100% positive it works that way. I think most sites are set up so that an order triggers a redirect to the receipt page. So a reload just reloads that receipt page. Going *back* triggers a "re-post form data?" confirmation dialog. But I'm out of my area of expertise, here. Regardless, the bottom line is that there is no way for us (speaking for the "front end") to tell Necko to do anything more than fetch it from the cache (if possible). See http://lxr.mozilla.org/seamonkey/source/netwerk/base/public/nsIChannel.idl#198 There is no "reload from the secret stash that's not the cache." I think, if the request is legitimate (which it very well might be), that it would have to be implemented, for the most part, in the http channel implementation, with a new load flag that we could then use to fetch the data in the various places where we want to reuse it (view source, save as, print, etc.). So, I'm resetting the component. At the least, Gagan will be able to comment on the subject with more authority than I possibly could. Re-summarizing, too.

Not actually sure if this is a networking bug. I'm wondering what piece of code actually hangs on to the document while it's being rendered, while the DOM is being modified by scripts, etc. Who owns the current document?... Please have a look at the attachment... De-nominating for beta3, removing milestone, but marking blocker, since it blocks two closely-related [nsbeta3+][dogfood+] bugs. This needs to be on somebody's radar, I just don't know whose.

forgive me if im stating the obvious with any of this. even worse, im not quite sure what DOM means, used in the context here. i'm taking it to mean the html equvilent of java bytecode (optimized half-rendered page), so if i guessed wrong there, i guess you should stop reading. we need three source caches. the traditional disk and memory caches, plus one to hold the source of the current page, regardless of the settings of disk/memory caches, and weither or not the server told us to cache the current page. seperately, there is the DOM cache (perhaps storing DOM for the last N pages would be a neat cpu/memory trade, but lets not stray). print/redraw and maybe character switching (17889) would ask for the DOM cache. view source/send page/save would ask for the source cache, which would fetch from the always-available current-page cache. the other alternative, the reserialization dr explained, seems like a pain, and error prone... maybe worth investigating in the future, i dont know.

jeremy: dom is the w3c standard object model interface for representing a document. > print/redraw and maybe character switching (17889) would ask for the DOM > cache. view source/send page/save would ask for the source cache, which would > fetch from the always-available current-page cache. that's one way of looking at it, but you have the right idea: use the object model where you don't need the source, to avoid reparsing; use the source where you do need it. > the other alternative, the reserialization dr explained, seems like a pain, > and error prone... maybe worth investigating in the future, i dont know. to your credit, it is definitely a pain if not well-thought-out beforehand. there is talk, though, of redoing our dom code sometime in the future (because we currently have separate code for html, xul, and other doms). this would be a place to include the reserialization information for xml/html. but, like i said, that would be for the future. for the present, we just need to keep the source of the current document in memory. perhaps if we're worried about bloat, we could choose only to keep it in memory when it's the result of a form submission... something like that.

Bug 17889 (to do with changing encoding), nsbeta3+, means you need to retain source at the *byte* level ...

For pages that dont give special no-cache settings, we behave the same as 4x But if the page had headers that disabled caching, then we will hit this bug in 4.x too. Would be nice to fix this. Since we dont have time, we will fix this for later release. The general idea of the fix is to keep all pages in cache (no-cache) and not get them from cache if they weren't supposed to be cached. Then add another reload policy that says get it from cache overriding these hints. Removing 4xp

We don't behave the same as 4.x here -- bug 20843 is what is the same as 4.x, that is, post results aren't cached, but in 4.x you at least were able to view the source or save the current page without refetching. that has nothing to do with the behavior of the cache. Bill Law summarized the bug best by saying "There is no 'reload from the secret stash that's not the cache'." This bug probably shouldn't even be in the Cache component. Also, adding dataloss (because it should have been there all along).

Is this the same as bug 50949? The bugs that depend on each seem pretty similar, and that one's fixed :)

As of the 09-27-00 build (give or take a day or two), the Advanced Debug option for "Enable Memory Cache" is gone, and documents do not seem to be getting caching in memory. Is this related to all this discussion of cache/no-cache here (bug 40867)? As I understand it, there are supposed to be two caches, disk and memory. If a document's HTTP header (set by the HTTP server) says "Don't cache", the document should not be stored in the disk cache. But shouldn't it still be stored in the memory cache, for re-use by Save As, Print, etc.?

atovar: there (are/should be) three caches, disk, mem, and current. current caches objects regardless of weither disk/mem are turned off, or if http sets nocache. I say are/should be, because i'm not sure of the status. if are is true, this bug is half closed.

Per law@netscape.com, this bug now also tracks the usage for View Source, i.e. "Make View source never load from the server". Thus nominating for mozilla0.9. See recent comments in bug 6119 for rationales.

When I type the url, "about:cache", I never see anything listed under memory cache, even when I have Debug / Enable Memory Cache set. Does this accurately show that the memory cache is never used? I was also thinking: Why is the memory cache disabled by default? Is this because we are relying on the operating system to provide sufficient in-memory caching for objects stored to the disk cache?

This bug is a *major* pain for Zope developers, since on server exceptions we embed a traceback in an HTML comment in the error page, so that it can be inspected with View Source. Couldn't the original text simply be attached to the root node of the DOM as a read-only attribute?

This seems to be an architectural problem, and a cascading one that's causing many other bugs. (With perhaps more bugs yet to be identified.) My understanding (correct me if I'm wrong) is that all these various functions (View Source, Save As, etc.) all ask for the page from the Necko library, which tries to fetch it from the normal memory or disk cache if possible, loading it from the network (or generating an error?) if it's not in the cache. I assume that the cache management subsystem is part of Necko, not external to it. Is this an accurate summary of how it works? Have I made any errors? Users and web developers alike have the same (reasonable) expectation -- that what they see in the browser window is what they'll get if they view the source, save it, etc. While that's often true, clearly it sometimes isn't. It can cause data loss and confusion, impede debugging of web applications, etc. I think the root cause of this problem is that these functions are asking Necko for the information, when Necko's function is to fetch data from URLs of all sorts. The very concept of "loading" the data to be viewed or saved is really nonsensical, because it's obviously ALREADY been loaded once! Forcing that "load" to come from the cache may improve the situation, but it doesn't solve the problem at all. It only masks the more obvious symptoms of the problem. I hate to suggest architectural changes, because they can often be problematic, needing to evaluate and fix any code that might contain implicit assumptions based on the old architecture. However, I believe the only solution that will ever fix this problem for good will probably need to be architectural in nature. Leaving Necko out of the picture for the moment, there should be an underlying relationship between the data structures that reflects the way we want to use them. For functions like "View Source" and "Save As", we want to make sure that we ALWAYS refer to the same source that is already being displayed in the window where the functions were triggered. This is a user requirement, and should be reflected in the data structures for maximum consistency and correctness. The simplest way to achieve this would be for the data structures associated with the window in question to contain or directly point to the exact source they are using. I assume they must already contain the DOM generated from that source, since that would be necessary to reflow the page if the window is resized. The simple solution would be to associate the original source with the DOM, and access THAT via the window's data structures for functions like "View Source" and "Save As", leaving Necko out of the picture entirely. This would necessarily be more correct and more reliable than asking Necko to fetch it from the cache. Older windows with outdated content would work as expected, despite the existence of updated content in the cache. POST results from form submission would work as expected. Clearing the cache wouldn't matter. Things would work much more as the user expects them to, and this would be a relatively small architectural change for a big improvement in correctness. The obvious downside of this simple solution is that it would use more memory; many pages would be saved that would duplicate information already in the cache. The better solution is probably a larger architectural change, separating the cache manager from the Necko library. After all, Necko depends on the caching functionality, and it would be best for these original source copies to be integrated cleanly with the existing cache -- but there's no good reason for Necko to be in the middle of functions operating on previously-loaded data. It would be cleaner to make the cache manager independent of Necko, while keeping Necko dependent on the cache manager. These functions on previously-loaded data could then depend on the independent cache manager without depending on Necko. This architectural change would make it impossible for these functions to unexpectedly reload the page from the network. An independent cache manager would need to be more flexible than Necko needs. (I don't know how flexible the current cache manager actually is; maybe it's already more capable than I imagine.) While Necko needs size-bounded in-memory and on-disk LRU caches keyed on URL information, functions such as "View Source" and "Save As" (AND Back/Forward history functions) need a reference-counted and/or garbage-collected memory management system, bounded only by available memory. When the LRU cache Necko needs happens to point to identical data, the cache manager could share the memory space. However, if Necko needs to replace its cache entry when content has been updated, or delete an entry, it must NOT change or discard any data still being referenced from, say, a browser window, if functions on previously-loaded data are to perform as expected. I've dodged a sticky issue. What should happen if the DOM is changed by Javascript code between the time the page is loaded and when the "View Source" or "Save As" function is called? There's good arguments for modifying the source to match the DOM changes, but this is clearly MUCH more difficult than tracking a copy of the original source. Even if this can be done, sometimes the user is likely to want the original source despite the DOM changes, so it should only be optional. (I just filed bug 63892 about this, since it's really an enhancement anyhow.)

cc to self

neeti, are you going to fix this, or should it be reassigned so some non-Necko person? It shouldn't be Futured without helpwanted for long. /be

This is something I'd be happy to volunteer for, if nobody else is ready to work on it. However, I can't guarantee how much time I'll have available to work on it, or how long it would take me. I'm not familiar with the Necko code or the current cache architecture, so I'd have to study it. I also don't know all the places it would need to be linked in. (Some pointers would be helpful.) My point is, I'm willing to work on this, but can't promise I'll get it done in any reasonable timeframe, if at all. If nobody else would be working on it anyway, then there's nothing to lose. If someone with more time available can dedicate it to this, I can look for other things to work on. Should I wait and see if anyone picks this up? Should I start working towards it just in case? Should I not have said anything until I had a patch in hand?

Wouldn't it be nice if it were possible to generate the original source accurately (including formatting) from the information in the dom? If that were true, then none of this would be an issue, we could just use the dom to regenerate the source without having to keep an extra copy around. Not throwing away whitespace, and finding a way to represent whitespace in attributes, would be a nice memory efficient solution to this problem, and would have the side benefit that people would be much happier with composer since it wouldn't always be reformatting their documents.

That seems like a possible solution, but it would require a considerable number of extensions to the DOM (some HTML-only, some perhaps XML-only): * remembering the original case of normalized attribute and element names and attribute values * remembering the whitespace in a tag: + after the element name + around the = separating attribute name from value + after each attribute * remembering the type of quotation marks around each attribute value * remembering the entire contents of any markup declarations (e.g., DOCTYPE) * remembering the delimeters of included marked sections * remembering the contents and delimeters of ignored marked sections * remembering which characters were originally entity references * remembering how every entity reference was terminated * properly remembering comment delimeters, since the W3C DOM Core doesn't fully describe SGML comment structure * remember whatever badly formed HTML (and perhaps XML, too, for view-source and save) is thrown at us * ...what did I forget?

This is an HTTP issue, not an HTML issue. Since right now things don't work right for *any* document type, it is premature to be discussing a complex optimization for one particular document type. And if you ask me, reconstructing the *exact* source from the DOM in a way that works 100% of the time, and is more efficient than simply keeping a copy of the original source, sounds completely hopeless.

It's surely not more efficient in terms of CPU time to regenerate the original source from the DOM, even if it's extended to represent insignificant (to the DOM) aspects such as whitespace. More importantly, it's a significant increase in complexity that begs for subtle bugs where you may reconstruct the exact source 99.9% of the time, but never 100% of the time. This would really work more cleanly and reliably at a lower level. The best solution is probably to save the content, byte for byte. Then it's straightforward to "recreate" the original source, because it's not a complex process of reconstruction from the DOM. Also, saving a byte-for-byte copy allows for changing the character set (bug 17889) or potentially reinterpreting the content as a different MIME type entirely. (Suppose the MIME type specified by the server is wrong? How about a function to reinterpret the data under a user-specified MIME type?) Also, if memory efficiency is a concern, and CPU time is considered a worthwhile tradeoff, simply compressing the source in-memory with zlib is a viable option. The DOM could also be compressed if it's not actively in use (i.e. history)...

Regarding documents being reformatted by composer, a more straightforward solution (especially if the byte-for-byte source is to be saved for this bug) would be to associate parsed elements in the DOM with byte ranges in the saved source. This is probably much simpler than annotating the DOM to retain case and whitespace information. Wouldn't that be as effective? (Modified elements could replace byte ranges in place -- fancier versions could try to guess some of the formatting conventions by heuristic tests on the surrounding source...)

IMO, viewing the source from the cache should be perfectly fine. If you need to debug, you should be able to control your server to ensure that the page can be cached or there should be a control in the browser to ignore "no-cache" HTTP headers. Please don't implement some drastic, complicated solution such as regenerating the source from the DOM or whatever. K.I.S.S.

I agree that reconstructing the source from the DOM is a drastic, complicated solution. However, I strongly disagree that simply reading from the cache is an acceptable solution. Not only do you have a problem with content that asks not to be cached, but also with older content that has been replaced in the cache with newer content (but may still be displayed in a window), form post results, missing data from manually clearing the cache, etc. The cache is inappropriate for the purpose, when the true intent of the functions (in the user's mind) is to operate on the EXACT data that they see in the window (or saw before). This is why I'm suggesting saving a byte-for-byte copy of the content, irrespective of whether that content should normally be in the cache. (Ideally, data thus saved would share memory with duplicate cache data for efficiency reasons...)

To reiterate: 1. You can't fetch the source from the cache, since it may not be there, or may have been replaced with a newer copy. I want the source for what I'm looking at, not what I'd get if I reloaded! 2. You can't reconstruct the source from the DOM, since the DOM may have changed, and it's terribly fragile to expect an accurate reconstruction even if it didn't change. A copy of the source needs to be associated with the window/frame/layer/whatever. It can be compressed. It may be able to be shared with the cache, but that's tricky. For now, why not just keep the compressed copy as an attribute of the appropriate DOM node, or history object, or whatever, and release it when the object it's attached to is released?

Speaking as someone who's already had to cancel a duplicate order caused by trying to save an order confirmation, I'd say pulling the data from cache is not sufficient.

You all have valid points. However, I think viewing the source is something that is only useful to developers, who will have the savvy to do what is necessary to look at the source (i.e. turn off the no-cache header in HTTP server or tell browser to ignore no-cache, don't reload page in another window, etc). 99% of ordinary users will never bother to view the source. Saving the current page is a different activity. Ideally, the page save function will offer the ability to save in a variety of formats, including HTML. However, I think that the data to save can just be generated from the DOM object and there is no requirement that the HTML (if the HTML format is chosen) be anything like the original source code. I agree that "Save" should absolutely never cause another request to be sent. In summary, I don't believe that "saving a page" is the same thing as viewing/saving the contents of the HTTP reply message that generated the page.

> Not only do you have a problem with content that asks not > to be cached, but also with older content that has been replaced in the cache > with newer content (but may still be displayed in a window), form post results, > missing data from manually clearing the cache, etc. A more flexible cache architecture could handle that (have a way of marking a mem-cached page as "window still visible, so don't overwrite until window goes away, but data not to be saved to disk cache" or some such). > 2. You can't reconstruct the source from the DOM, since the DOM may have > changed, and it's terribly fragile to expect an accurate reconstruction even if > it didn't change. If the DOM has changed, then so has the display in the window; don't you want to see the source for what you're really looking at, rather than what you loaded some time ago? > to associate parsed elements in the DOM with byte ranges in the saved source. That would be very useful for composer. The parser would have to do the association, though; trying to reconstruct it later would be error prone and probably CPU intensive.

cc'ing gordon who working on the new cache.

Nice summary, Evan. Personally, I agree with everything you said in it. Ken, bug 6119 was specific to viewing the source; this bug applies as much to saving a page as to viewing the source. You seem to be suggesting that saving a page generated from the DOM would be sufficient. As a user, I would NEVER want the browser doing this. Even ignoring the readability of the code, if the browser is just saving what it *thinks* the source represents, it invites errors to creep into saved pages that weren't in the original source. A bug in the HTML parser could cause a buggy page to be saved. (What if it didn't render properly and the user is saving the page in order to view it in another browser?) Unsupported HTML tags (or attributes) would be discarded. Also, Mozilla itself could fail to view the saved page as before, if there's a bug in the generation code. While regenerating the HTML from the DOM is possible, it's complex and risky, and offers little advantage over the straightforward solution of saving the original source in some fashion...

Akkana: See my comments (for this bug) of 2000-12-28 14:39 for some of my earlier thoughts about how an independent cache manager could meet the needs of Necko's LRU cache and this sort of caching of historical content simultaneously. As for DOM changes, I don't think it's cut-and-dried. Probably you'll want to see the original source sometimes and the source as modified by DOM changes at other times, depending on the circumstances. As a user, I certainly wouldn't want to lose the ability to save a byte-for-byte copy of the original source, even if the modified DOM could be used to generate a more-current version. Both options would be nice, but I believe the original source is critical... As far as the parser needing to associate the byte ranges within the DOM, that was implied. It would be senseless to try to figure that out afterwards; the parser would definitely need to do that part of the job. Neeti/Gordon: I guess I'm glad someone is going to be working on this, but I'm a little disappointed at the same time. I was hoping to take a stab at it; it sounded like fun! (But I wouldn't want it held up waiting on me, since I can't guarantee I'd find the time -- I don't have the luxury of being paid to hack Mozilla full-time...) If a wild hare makes me play with it anyway, I'll let you know. (But only if there's something worth seeing!)

Actually, when I look at the source, 99% of the time I am interested in the source as it came from the server. This is certainly the case when I try to save a page. If the DOM has been modified by client-side scripts, I would want to be able to save the original source and scripts and not have extra HTML that's generated from the modifications to the DOM. In an ideal world, we would have both options: 1) looking at the source of "what we see now". This would need to be generated from the DOM. 2) looking at the original source. This is most likely impossible to generate from the DOM because it does not necessarily correspond to the DOM (elements may have been added to the DOM through scripts). It looks like the original purpose of this bug was to cover option #2. Should we open a separate bug on option #1?

I wanted to disagree with KenW's comments. I think that it is not a safe assumption to make that a developer should be able to frob the cache directive if they're debugging server-generated pages. IMO, view-source should function independently of any cache directives. Furthermore, when I save a document, I want the original source in all cases that I've encountered so far. In my experience, I end up wanting to save a page so that I can do a diff against an earlier saved page. I think it's dangerous to think of saving a page as a non-developer function. Both viewing the source and saving the page should be lossless.

We have been working on some design changes to the cache that should make it easier to address these problems. I hope to push a proposal out to mozilla.org in the next few days. The decision to keep the source around for currently viewed pages doesn't belong to the cache, but some of the changes we want to make will make it easier for current pages to share data with the cache avoiding unnecessary duplication.

Adding self to CC.

*** Bug 67781 has been marked as a duplicate of this bug. ***

*** Bug 68677 has been marked as a duplicate of this bug. ***

I don't think 99% of the users never bother to view the source. Although HTML was not meant for such a purpose, there are many people out here that generate pages at hand, and thus want access to the source whenever requested. That includes Javascript that can modify the code -both unaltered and modified source are worth interest in that case, but the unaltered one is the more important. When saving a page to the disk, the original page need to be saved. The changes in the DOM will operate when loading the page in a browser, executing whatever javascripts etc. there are in the page. If the settings of the browser have changed (disabling Javascript...), so will the displayed page. I consider this as a matter of consistency in the browser rendering. A question I have is whether or not keep images, embedded objects of the page in memory or not. Suppose I want to print a page: using the pictures from the DOM would be OK, wouldn't it? There is no use in refetching them

cc'ing pavlov and saari for their view on images and printing. Short answer: sometimes (often?) the images need to be redecoded at a different resolution for printing.

Yeah, pulling the unmodified data out of the disk cache and redecoding it when we need to print would be ideal for images with greater than screen resolution dpi.

jido@respublica.fr don't think ... <people> never bother to <act> is a double negative. While it is valid English, people tend to worry that the author is confused. the literal meaning is: believe <people> <act>. Saving objects with the page is the subject of other bugs. printing should use the current data. WYSIWYG

Printing should produce a WYSIWYG replication of what you see on screen in the highest possible resolution, and that will be bounded by the printing device or resolution of the image, whichever is smaller. We're not printing a 72 dpi image if we have a 300dpi copy sitting around.

so long as we know that the image @300dpi is related to the one @72 dpi and not something that could be entirely different.

Yes, both/all resolutions are derived from the same http get of the compressed data.

Gordon: do we have any progress on this bug? I'm currently working on a set of webpage scripts that have form POSTs all over them and debugging them is almost impossible due to this bug. It's driving me up the wall :(

I'm not sure if the problem I was encountering was related to this. When I was using some sort of web account and making operations and submitting forms with cookies, mozilla sometimes re-post my login information when the page direct me back to the main page (which should be cached), and the server thought I was attempting to multiple login and refused my request.

Cache bugs to Gordon

Is the new Cache Manager going to fix this? It'd be a real handy thing to have, as a developer. e-commerce and other apps where reloading will totally screw you could use it, too.

The new cache itself won't fix this, but will enable the docShell to hold onto references to cache entries for the current page ensuring they won't get fetched from the server again. I'm not sure if docShell will take advantage of this however.

Resetting milestone from "Future" because at the time it was set it just meant "post netscape6 release", and we have milestones up to mozilla 1.2 by now.

Any follow-up on this one now that the new cache is backing in the tree?

From what I've learned about the new cache, we now have a way to keep the source for currently viewed pages in the cache using the hard reference capability. Just keep the hard reference around. This capability is is not used yet, however.

If I understand this correctly-- at this stage this is just a docshell issue. ->docshell owner then.

I'm slightly worried about letting the cache handle source storage. Consider this: user visits a dynamically generated page, and leaves the window open. A little later, they visit the page again in a different window, getting different results. Now the "same" URL is open in two different windows, with two different source texts. Can the cache handle this correctly, or will both windows seem to have the source text that was loaded more recently?

The cache can handle this correctly *if* the docShell holds strong references (cache tokens) to the items in the cache that it used to render itself. If docShell doesn't hold the strong references, viewing source on either window would use the latest version (possibly fetching a new third version from the net).

*** Bug 78740 has been marked as a duplicate of this bug. ***

Yes ... and it's much more efficient to hold it in the cache because then you have only one copy of the document ... why keep it around a 100K document in your current page structure for a long period of time if it's already sitting in the cache? Here's the sequence of events you're talking about: 1. User surfs to http://www.yahoo.com. - http://www.yahoo.com comes down from the network. We grab a copy from the cache. - We grab a hard reference to that copy immediately. - We keep that hard reference around. This means the cache agrees to never get rid of it. 2. The page expires from the cache. - Since the entry has a hard reference, it stays around in memory but gets Doomed. Thus it cannot be searched. - Note that this is also what will happen if the page is set not to be cached at all. 3. The user surfs to http://www.yahoo.com in another browser. - Since the entry is doomed, the search through the cache does not find it. - http://www.yahoo.com comes down from the network. We grab this new copy from the cache. - We grab a hard reference to that copy immediately. - We keep that hard reference around. This means the cache agrees to never get rid of it. - Both copies of the page are in memory now; each hard reference refers to its own copy. 4. User closes the first browser. - The browser lets go of the first hard reference. - The cache realizes that the entry is doomed and so immediately dumps the document from memory. 5. User surfs somewhere else in the second browser. - The browser lets go of the second hard reference. - The cache entry is not removed yet (because it's not ready to expire) but when it expires it will be removed normally now instead of being Doomed.

Exactly!

Gordon, can the cache token be passed around? For example, view source loads in a different docshell than the original document (it's in a different <browser/> element...). Is there a way to pass the cache token to the new docshell? And are such things scriptable?

Yeah, they could be passed around, but some recipients may not have enough context to be able to use them. We would definitely want to enable their use by view-source. The sooner we can work out the details of what's required the better. I believe the http channel is scriptable (that's where you get and set cache tokens). There may still be a bug open to implement the setting of cache tokens on http channels; Darin would know for sure. We had been waiting to get an actual client for them first. Take a look at nsICachingChannel.idl. Cache tokens are just an opaque nsISupports that happen to contain an nsICacheEntryDescriptor, so a http channel knows how to use them to rehydrate itself.

this bug depends on a bug that is marked future. I'm reflecting that in this bug's milestone.

Considering that this is a correctness/dataloss bug with 36 votes that has been targetted to mozilla0.9.1, and that the reason given for futuring bug 72519 is "there are no consumers for it", wouldn't it perhaps be better to ask whether the futuring of bug 72519 could be re-evaluated than to just future this bug which obviously a lot of people consider to be very important... It's also not as if depending on a futured bug is an absolute impediment to this being fixed; I notice that 3 of the bugs "blocked" by this bug, and one other bug "blocked" by bug 72519, are already fixed anyway.

This no longer depends on bug 72519 anyway. The function we need is GetCacheToken() and that is implemented now. The bug covers both GetCacheToken () and SetCacheToken(); it is the latter that is not yet working, and that is not necessary for this bug. Everything necessary to fix this bug is now in Mozilla. There is no need to change the milestone unless the developers cannot do it in time.

clobbering milestone and killing dependency.

Question: should the browser keep references to the current page's images also? It seems like it should. Also if the current page contains frames we need to keep around references to the frameset as well as every single frame. Otherwise this fix is not accomplishing everything it should--namely a perfect representation of the current page for Save, View Source and Print.

Setting cache tokens probably *is* necessary for this bug to be fixed reliably. Simply holding a cache token won't guarantee that what you request from HTTP is the same thing that you're holding the cache token for.

OK, my apologies. Then there's something I don't understand. When I read bug 72519 I naturally assumed that GetCacheToken got you the nsICacheEntryDescriptor (hard reference) referring to the document. Indeed, reading the code, that seems to be the case. Hmm ... perhaps the hard reference is not always set up by the time you call GetCacheToken()? Is SetCacheToken() supposed to place the document into the cache and set up the hard reference? Or perhaps does http place the document into the cache automatically but SetCacheToken() has to be called to keep and hold a hard reference to the document?

GetCacheToken() will allow you to keep an entry in the cache, but only SetCacheToken() will allow you to recreate a channel to that entry. Otherwise you have to create a channel with a URL which may or may not be what the cache token refers to (the entry has been doomed).

Oh, I see. So what you're saying is we will do this: * User types in URL - Browser creates HTTP channel - Browser has HTTP channel retrieve content from URL (possibly through cache) - Browser grabs hard reference from HTTP channel and keeps it around (GetCacheToken) - Browser destroys HTTP channel * User hits View Source - View Source gets cache token from Browser - View Source creates HTTP channel - View Source has HTTP channel retrieve content using hard reference (SetCacheToken) - View Source destroys HTTP channel Why not change the second part to this: * User hits View source - View Source gets cache token from Browser - View Source grabs content directly from cache token Then you don't need SetCacheToken(). HTTP channel seems like an unnecessary extra step in this case. Maybe it's impossible to get a stream out of the cache, and that's why it's done this way?

I think this also causes us to suck on the IBENCH tests.

No that's bug 61363. The IBENCH tests we're looking at aren't doing printing, saving, or view-source. This bug is for operations on the currently displayed page, so they don't result in refetching from the net. When the user wants to print, save, or view-source for the front-most window, they don't expect to get different results from the net.

Do you think we can get this resolved in time for mozilla0.9.1?

BTW, another situation when we should use the current version instead of refetching is the "Send page" function (which does not work correctly even in Netscape 4). With "Send page"not only you might end up loosing the data, you may also end up e-mailing something very different from what you thought you are e-mailing...

-> 0.9.3

*** Bug 79758 has been marked as a duplicate of this bug. ***

bug 83792 - same cause, more unwanted effects. Fix this NOW, please. (Sorry for the spam)

a possible (minimal) solution: toplevel page is loaded... before the necko channel goes away, QI to nsICachingChannel. then call nsICachingChannel::GetCacheKey. store the cache-key someplace. when you want to reload the document, just create a new necko channel, but before calling AsyncOpen, QI the new channel to nsICachingChannel, and call SetCacheKey, passing the stored cache-key to the channel. also, be sure to call SetLoadFlags(nsIRequest::LOAD_FROM_CACHE) on the necko channel as well to prevent the usual cache validation. the cache-key is used, for example, to distinguish the results from different POST requests on the same URL. this solution does not guarantee that data will not be fetched from the cache, but it does do the "right thing" regardless of whether the cache is present or not, etc. and, for file->SaveAs and view-source it will almost always avoid refetching from the net.

Folks, could someone please step up and make a damn architectural decision here? over a year of thinking about this issue in this bug alone and we don't seem much closer to a solution. There's been a lot of good discusions and ideas kicked around, and a number of possible solutions proposed, but we still don't appear to have anyone setting a stake in the ground and saying "do this." (or better yet a group of core moz folk that can agree on single solution.) I'm currently loosing a $100 argument with buy.com over a duplicate order that resulted from printing a copy of the order summary. At this point mozilla is unusable for the average user's e-commerce (at least with sites as poorly engineered as buy.com). I've had to resort to another browser for any form of online purchase.

putterman: were you asking me about ecommerce and multiple orders?

Radha, what's the session history perspective on this?

PDT+ per selmer's request.

Session History comes into picture only when you hit reload() which uses the VALIDATE_ALWAYS loadflag. Session History however stores the cachekey for the current page in mOSHE. This cacheKey is passed to nsICachingChannel when the loadtype is LOAD_HISTORY/RELOAD_NORMAL/RELOAD_CHARSET_CHANGE. If the solution to this bug is to do a similar thing for view-source/printing, then I think the current setup in nsDocShell::DoURILoad() can be extended to meet the needs here.

I'm not seeing any problems with printing POST results. On both the trunk and the 0.9.2 branch when printing the results of a form POST we *do not* reload from the network (and do another POST)... Are there any sites that exhibit this problem when printing? (ie. reposting when printing...) If not, I suspect that this is only a SaveAs issue... Because the printing code deals with documents in a completely different way... -- rick

The attached patch implements Darin's suggestion for passing in the cache key to the request generated by a Save As... operation. The patch attempts to load out of the cache. If there is post data and the stream isn't in the cache, it brings up a dialog asking if the user wants to repost. The Save As... menu item works as expected with the patch. I saw a problem with the Save As... context menu item, but it seemed as if the JS in nsContextMenu.js was passing a bogus false value for doNotValidate. Still investigating, but the patch should be set to go in as-is pending r/sr's.

Rick, was that a recent change in printing? I know it used to go back to the network (for get requests anyway) in pre 0.8 days... I had gotten burnt a few times printing the results of some internal tools we use and getting hard copy of the input forms instead. with current trunk builds here's the matrix from a quick test I just did: (pre vidur's patch which I just mid-air colided with... building with it now.) POST GET --------------------------- print OK OK view source OK OK save as BAD! OK hmmm.... one in six... russian roulette anyone? (p.s. after four months, mention of the state attourney general last week seems to have finally solved the discusions with buy.com)

with that patch my test case is 100% OK.

Chris, how about "File -> Send page" (bug 86261)? I haven't checked it yet, but I am afraid it will add a bullet or two to your "Russian roulette".

POST GET --------------------------- print OK OK view source OK OK save as OK OK send page BAD! OK

http://ntiaotiant2.ntia.doc.gov/test/mozillatest is a much better test case from cweiss@iname.com on bug 55583. summary for those not on that bug: save as: "successfully" re-requests! (number incremented) view source: fails completely (server error, suspect GET instead of POST) send page: fails completely (server error, ditto) print: success :)

vidur/adam - any update to this PDT+ bug? Pls update status whiteboard. It seems from Chris Abbey's comments that the patch from Vidur works ok? If so, can this patch get r and sr and be checked in? It looks like the patch will fix Save As, but not Send Page. Perhaps that is a separate bug.

Latest patch looks good to me so r=adamlock@netscape.com. I think Send Page should be new bug.

Well, according to last Chris Abbey's test results, _all_ methods currently re-request, so there is still a need for some universal method of avoiding re-requesting. But once such method is implemented, making use of it becomes separate bugs - bug 55583 for "view source", bug 84106 for "save as" and bug 86261 for "send page". IMHO, what this all means is that the "save as" patch belongs to bug 84106 and not to this bug.

lchaing: my append on 6 July was with the trivial test case I attached, the append on 7 July however was with the more realistic testcase at the URL. (I'm not sure the difference, aside from a cfm script vs a php script, but it clearly demonstraits one). As a result I wouldn't hold much faith in the value of my append from 6 July if I were anyone.... ;) Aleksy: as rick pointed out, and the URL proves, print works fine... but then print is also a completely different beast in that it really doesn't *need* the original html, the dom is fine. Otherwise I agree with your take on this bug 100%.

The patch ensures that for Save As we first attempt to get the data from the cache and, only if that isn't possible (e.g. someone explicitly clears the cache), we put up a dialog asking if we should repost. The same pattern should be used for the View Source and Send Page cases as well, though the fixes for these two will require changing different code paths. I got a verbal sr=rpotts@netscape.com, so I'm inclined to get this into the trunk for some bake time.

Vidur, exactly, and this is why the patch belongs to bug 84106. What this bug is about is creating a mechanism that would ensure that if you are viewing a page, then you also have a cache takon that will ensure that the source will be in the cache if you need it. This patch is doing the right thing (hopefully) and is addressing a related issue (as covered by bug 84106), but it is not addressing this bug.

Vidur, does that patch also fix "Save Frame As..."? It also uses the savePage() function, but I'm not sure the history entry you're getting has the post data for a frame... (in fact, based on my testing it seems not to).

Also how about "View Frame Source"? I was having trouble with that not working correctly for POSTed data with 0.9.2 today, which is what reminded me.

"View Frame Source" is what _I'm_ working on (bug 55583). That was my 'testing'. If there's a way to get the postData and cacheKey associated with a document in a frame, I have yet to find it.... Hence my previous comment in this bug.

Not sure if this is pertinent but thought someone should check. There have been a number of comments posted about "postData" does this need to include/does include the cookie data at time of posting? Obviously if you were refetching the page from the network that would then be important, but is it important for re-obtaining the page from the cache? or any other location? For instance you go to the same page URL+Post data, but since the submission has already been posted once, now the source could change based on it being the second post. So then seemingly identical "posts" could be different based on cookies, or the current time, etc, but would both appear to be the same in the cache, does the second post overwrite the first cache? should it? Glad to see this bug finally getting some heavy attention....

docshell member mOSHE has the cachekey and postdata for the document. THis applies for subframes too.

Radha: is that member somehow scriptable? The problem is that we're working in JS here... Wiggins: The important thing is the combination of cacheKey + postData. The behavior on load is then as follows: if (cacheKey && postData) { // try to load from cache using cachekey. If that fails pop up a warning // dialog before requesting from the network } else if (cacheKey) { // load from cache but do not prompt before re-requesting } else { // just load }

mOSHE is a private member of nsDocShell, not available from JS.

is all or parts of this on the trunk or branch? what's left to do? can someone summarize and project an ETA?

Here's what needs to happen as far as I can tell: 1) An API is needed to get from a docshell (any docshell, not just the root of the docshell tree which has the session history attached to it) the information needed to recreate it (load from history or reload from the web with the same post data). At the moment this would be a cache key and a post data stream. These could be gotten as themselves, in the form of an nsISHEntry, or whatever. 2) A method needs to be added to the nsIWebNavigation interface that takes the information gotten in step #1 and loads a page using it. This could be similar to the current nsIDocShell::loadURI or something along those lines. Once that's done, this bug can probably be marked as fixed. The dependent bugs can then handle using these APIs to make save page, save frame, send page, view source, view frame source, and so on work correctly. Ccing Rick and Jud who've been talking about this a bit lately, since they'd be the ones who could provide an ETA (and corrections in case I've gotten something wrong).

What's up with this bug? No current status and time is almost gone!

At chofmann's request, I'm posting this summary of status. Currently, the following work correctly if there's post data associated with a page: - print page - print frame - save page The following still don't work correctly if there's post data (I believe they will always refetch with a HTTP GET): - save frame - view page source - view frame source - send page For NS6.1 RTM we have the following options: 1) Disable the latter set of operations for pages that have post data. To do this correctly, specifically to even be able to determine whether the document in an individual frame has post data associated with it, we still need some amount of infrastructure (as opposed to chrome JS) change. Our options include: a) Expose the currently private nsISHEntry member of a docshell. b) Expose a mechanism for getting information about post data at the nsIWebNavigation level for both top-level content windows and individual frames. Once one of these changes are made, we'd need to make JS changes to check for the existence of post data before completing the corresponding operation. 2) Come up with a complete fix. Boris Zbarsky's description of such a fix (or at least the infrastructure for it) is on the money. We'd have to: a) Expose a mechanism for getting the post data at the nsIWebNavigation level (as in 1b). b) Allow for the post data to be passed in as an optional parameter to nsIWebNavigation::LoadURI (bug 46870). c) Either expose the notion of a cache key in a similar manner as the post data stream or hide the cache key completely. The latter option would require us to create an internal mapping between URIs and cache keys (including URIs that are not "officially" in the cache, such as those that have post data). Once these changes are made, we'd also need to make JS changes to retrieve the post data and pass it into the load methods used for the different operations. 3) Live with the status quo Rick Potts is looking at the infrastructure changes for 1 and 2. He will first work on the infrastructure changes needed for disabling (probably option 1b/2a). He will then check in an existing patch for 2b (see bug 46870). The ETA for these changes is 7/18. At chofmann's request, I will open bugs for making the chrome JS changes necessary to disable the features for pages with post data. If Rick can't get the complete fix done by 7/18, chofmann will find owners for the chrome bugs.

By "parts working" I assume you mean there was a checkin on the branch. If so, I'm leaning toward option 3.

A few comments on what vidur wrote: > 1) Disable the latter set of operations for pages that have post data. This would be a branch-only thing, I presume? On the trunk we would keep working toward option #2? > The latter option would require us to create an internal mapping between URIs > and cache keys This may not work so well for view source... Consider the page at http://www.mozilla.org. To view its source we load the URL view-source:http://www.mozilla.org. The protocol handler for view-source: creates an HTTP channel for http://www.mozilla.org and loads it. _But_, we want to use the source from the original URL. So we want to take the cache key and post data corresponding to the http: url and load the view-source: URL with those. This does not mean that the cache key has to be exposed, it just means that there needs to be a way to load a url of the form view-source:whatever using the information for the "whatever" URL.

>This would be a branch-only thing, I presume? yes > On the trunk we would keepworking toward option #2? yes The most troublesome remaing problem for most users would be "send page", I could see users completing an ecommerce transaction and then attempting to send the page to themselves as a confrimation of the transaction. bug over to rpotts since he is working the infrastructure changes that are now the focus of this bug.

In order to make this work correctly, 3 different API changes must be made: 1. Provide a 'postData' attribute on nsIWebNavigation to allow JS to access the postData. 2. nsIWebNavigation::LoadURI(...) needs an additional 'postData' argument. 3. The underlying necko 'cache key' needs to be hidden from non-networking APIs. Clearly, we can't make all of these changes on the branch :-) #3 alone requires a fair amount of work - and darin is currently gone :-( So, here's my propsed plan... ----------------------------- For the branch I can make API change #1 and expose a postData attribute to JS. This will be enough to allow all JS implementations to 'disable' if postData is present. Fortunately, this API change is *very* small and should not introduce regressions... it does not change any existing APIs and the only consumers will be the new JS code written to disable the features. For the trunk I can land API change #1 and #2 and start working on #3... Once all three changes have been made, we will be able to 'correctly' implement these features in JS... Does this sound reasonable to everyone? -- rick

sounds like a good approach to me.

Okay, I'm an outsider here, but I'm slightly worried about the #3 approach. Consider the following scenario. I'm assuming a page that simply provides an incrementing counter every time it's hit, but the same argument applies to almost any page that can give different results on successive hits with the same querystring and/or post data - anything that, for example, is time-sensitive (eg frequently updated news sites), relies on a backing database on the server which changes state, is content-negotiated (if the user changed their language prefs) or cookie based, etc etc etc. User opens page in window 1. Page is fetched from server, returning "1". Cache entry #1 is created, containing "1". User leaves this window open on this site and goes to work in another window. User later opens same page in window 2. Page is fetched from server again, returning "2". Cache entry #2 is created, containing "2". Cache entry #2 has the exact same URI, and also the exact same post data (if any) as #1. User goes back to window 1 and selects "View source". If the "View source" implementation only has the uri and post data to go on, it has no way to distinguish between cache entries #1 and #2, but they contain different things! Since #2 was created later it will probably get that, which is wrong. Surely a better way to go here would be to expose the cache key on the document directly (as an opaque pointer that you can't do anything with except load, of course) and use this for "view source", "send page" etc. Relying on the URI plus post data seems like a step backward. Am I missing something?

I have the same concern as Stuart. More specifically, hiding the cache key should still somehow allow loading url "view-source:foo" and getting the source of "foo" from the corresponding window from cache.

>Does this sound reasonable to everyone? After the long wait, sounds like music to the ears :-) Although the reasonning so far throughout the bug has been that the cache key is "key" to fixing the bug as sballard has illustrated.

A second URL fetch performed with identical post data will not retrieve the same cache entry unless a "cache key" or "cache token" (from the original channel) was provided to the HTTP channel. The original cache key can not be recreated given an identical URL and identical POST data. I'm not clear on why we want/need to hide the cache key.

One of the issues is that there is not a "single" cache key associated with the document - correect? there is a cache key per URI loaded for the document. This means that to be "correct" someone needs to hold onto *all* of the cache keys - including the ones for linked CSS and JS... Holding onto the cache key for the document URI will definately make things "better" but i don't believe that it is sufficient to be "correct". If these assumptions are correct, then I believe that someone has to hold onto *all* of the cache keys... And i'm not about to add an nsVoidArray argument (for them) to LoadURI(...) :-) I believe that we need to come up with a "correct" approach.

rick, when can you get: 1. Provide a 'postData' attribute on nsIWebNavigation to allow JS to access the postData. on the branch? lets make that happen and then we can take the pdt+ off the this bug. vidur, can you add the bug numbers for the open bugs for making the chrome JS changes as a reference in this bug? those also need to get pdt+

No issues with the branch which can have its quick fix -- but if it takes adding a nsVoidArray argument for the correct fix later on the trunk, then there are 59 votes and XXX persons cc:ed here for that :-)

Can any netscape people comment on how 4.x handled this? (I never saw 4.x get it wrong, although I can't say I tried very hard). A single cache key would work fine for view source, at least. Is there any way that the document itself can hold cache keys to the other things that were loaded by it? That way everything gets cleaned up through normal refcounting and we only need the single cache key to the document. Perhaps the extra VoidArray should be added to the CacheKey object?

stop it... you all are hurting me :-) with nsVoidArray example rick meant to say that this is a bad idea... we clearly need to work a better solution than having to keep all the cache-keys (or any) Lets stay focussed on resolving this for both short term and eventually with a better solution past this release. And rick: next time you suggest something like a nsVoidArray of cache keys (without sarcasm/kidding tags around it)... I am gonna come looking for you with my nerf gun :-)

gagan, how many smiley faces do i need? i always thought that one was enough :-) but perhaps when the cache is involved i need a few more :-) :-) :-) -- rick

once again i seem to have ignited a hot topic :-) My main concern with exposing a cache key via the nsIWebNavigation API is that i do not believe that it is sufficient to guarantee that *all* content will come out of the cache. I believe that we may need some other mechanism in order to "get it right" 100% of the time. So, rather than put the cache key there "for now" (and realistically never revisit the issue until another PDT+ bug turns up) i think that we should hold off and consider what the correct solution is - for the trunk. I'll try to attach a patch for the branch (really soon) which exposes a read-only attribute for the postData on nsIWebNavigation... At least this will allow us to *not* do the "wrong thing" :-) Once the branch work has been done, i'll start looking into what the "right thing" is for the trunk... So, Gagan, get your nerf gun ready ;-)

I've just attached a patch which adds a read-only attribute to nsIWebNavigation allowing access to the postData (if there is any...) So with this patch, given an nsIDocShell you can QI to nsIWebNavigation and check for postData. When dealing with frames it is important to get the nsIDocShell associated with the particular frame in order to insure that you are getting the right post data :-) -- rick

As far as that goes... we need to get frame docshells in response to context menu actions. At the moment when a context menu comes up we have the document involved (this.target.parentDocument) and the window involved (document.commandDispatcher.focusedWindow). Neither of these allows easy access to the corresponding docshell. One can walk the docshell tree and for each docshell compare nsIDocShell.document to this.target.parentDocument till a match is found (I have a function that does this lying around). Is there a better way?

from an nsIDOMDocument you can do the following to find the corrosponding nsDocShell: QI(nsIDOMDocument) -> nsIDocument nsIDocument->GetScriptGlobalObject(...) nsIScriptGlobalObject->GetDocShell(...) and you're there!!!

nsIDocument is not scriptable... I'm working in JS.

In terms of the question of how to fix it "right", my suggestion on 2000-12-28 was to make a separate cache manager instead of trying to adapt Necko to do it. This still seems cleaner to me; am I alone in thinking along these lines?

Stuart, as far as I remember 4x indeed did it right most of the time, but not for the "send page".

The bugs to disable view source and send page on the branch are 91341 and 91342.

The PDT+ on this bug on the branch is *just* for the patch necessary to make bug 91341 and 91342 possible (we've just PDT+ed those bugs.) Can this be checked in today?

Could someone review and superreview Rick's branch only patch? thanks!

Rick, we really need a way to get the docshell of a document from js. Do you know of one?

r=bbaetz for the branch. Having the input stream implement nsIRandomAccessStore doesn't help js, because that interface isn't scriptable, and we need this for js. For the branch, blake only needs to know if the data is post data or not, so thats OK. It needs work for the trunk, and possibly a different solution.

The input stream is rewound at http://lxr.mozilla.org/seamonkey/source/docshell/base/nsDocShell.cpp#4526 and any loads that use the nsIWebNavigation APIs should be passing through there... Code that creates channels directly instead of using the nsDocShell stuff will have a C++ part that can rewind the stream.

Blake super-reviewed Rick Pott's branch only change and also checked it into the branch. PDT shd remove PDT+ from this as it is no longer a stopper for the branch.

hey blake, jst and I talked about the best way to get an nsIWebNavigation from an nsIDOMWindow... and as you know, currently, there is no easy way. The best solution we came up with was to an nsIWebNavigation as one of the interfaces that you could get via GetInterface(...) this would require two small changes in nsGlobalWindow.cpp 1. Add nsIWebNavigation to the GetInterface(...) implementation. 2. Change the ClassInfo flags so nsIInterfaceRequestor was visible to JS via XPConnect. Let me know if you think that these changes are necessary for the branch... I think that we will definately want to do this on the trunk... since we need a way to map from an nsIDOMWindow to the embedding interfaces... -- rick

Rick, thanks for the info. Last I heard was that we could live with it being a problem in View Frame Source on the branch. Lisa, does PDT want to pursue fixing it for view frame source too? [removing PDT+; there's nothing that needs verification here anyways]

No way Netscape 4x did it right most of the time... How many times did I curse Navigator trying to fetch the document again in offline mode for a printing or for a view source? And what about when View Source provided the source of the error page instead of the displayed one? Maybe it got better with 4.75 or 4.76 but it was a very late effort.

Netscape 4.x doesn't do it right. I just tested 4.76, and it's very broken. I'm sure 4.78 is much the same. I've uploaded an attachment with the Perl code of the CGI that I used to test with; it sends back the current date/time, the method used to call the CGI, and a link to itself to trigger the GET method, and a submit button to itself to trigger the POST method, so both can be tested. If you wait at least 1 second between tests, you can tell whether or not the CGI was reloaded. Under 4.76 (Linux), here's what I found: The GET method will perform the "View Source", "Print", "Send Page" and "Edit Page" functions, but ALL of these will re-fetch the page and run the CGI again, since the content has expired from the cache. (If it's in the cache, it will probably use that for some or all of these functions, but that's not correct behavior either.) All functions using the GET method will use the page, but none of them use the copy being displayed in the window. The POST method is even worse. "Send Page" and "Edit Page" both reload the page with a GET request (not POST), running the CGI again. "Print" fails completely, sending nothing to the "lpr" process, which returns the error "lpr: stdin: empty input file". "View Soruce" will bring up a window, but the source displayed is that of the usual POST error message: <TITLE>Missing Post reply data</TITLE> <H1>Data Missing</H1> This document resulted from a POST operation and has expired from the cache. If you wish you can repost the form data to recreate the document by pressing the <b>reload</b> button. Basically, NONE of the functions in Netscape 4.x that should be using the original source are correctly doing so. When they manage to, it's only because the content happens to be in the cache, and it could expire from the cache or be replaced with a different version at any time. For content that expires right away, the ONLY thing that works on the original data is cut & paste with the mouse into another window. That's all. Netscape 4.x doesn't even come close to getting it right.

As rpotts explained earlier, I have attached a patch to nsGlobalWindow.cpp and nsDOMclassInfo.cpp that exposes nsIInterfaceRequestor to JS. I need this functionality for some other purposes too.

Radha, don't expose new interfaces on the window object in JS unless it's absolutely required, in this case it isn't. In stead, allow scripts to QI the global object to this interface when this interface needs to be accessed. This means that the window object in JS won't have a getInterface() method on it, in stead you haveto do window.QueryInterface(Components.interfaces.nsIInterfaceRequestor).getInterface(...), this way we can support new functionality w/o polluting the global namespace on web pages. I'll attach a patch that does this part, your change to nsGlobalWindow.cpp is fine, but don't check in the nsDOMClassInfo.cpp change in your patch. With my patch (once tested n' all that) you'll have sr=jst

Re: jst's patch, I'm assuming that the new inclusion of nsIXPCScriptable::DONT_ENUM_QUERYINTERFACE in the scriptable flags for Nodes, Arrays, etc. is not going to break anyone. sr=vidur

r=rpotts on jst's patch to nsGlobalWindow to expose nsIInterfaceRequestor via an explicit QI(...) and r=rpotts on redha's patch to add nsIWebNavigation to getInterface(...) once the DOM_CLASSINFO_MAP_ENTRY(nsIInterfaceRequestor) is removed...

Vidur, the scriptable flags remain unchanged for Nodes, Arrays etc. (they already have nsIXPCScriptable::DONT_ENUM_QUERYINTERFACE set), only the flags for Window are changed.

The latest patch consolidates the previous 2 patches (patch ids 43271, 43347) and a patch given by jband privately for a assertion problem. This patch securely exposes nsGlobalwindow interfaces to JS. will be checkin this to the trunk. Reviews were given to individual patchlets.

I don't quite see how printing comes into this. Dosn't print work on the (currently displayed) DOM? IMHO it should, printing should be wysiwyg.

*** Bug 93157 has been marked as a duplicate of this bug. ***

*** Bug 93890 has been marked as a duplicate of this bug. ***

*** Bug 94417 has been marked as a duplicate of this bug. ***

What's the latest on this? They want me to disable edit page/send page/view source again for pages with postdata on the trunk unless this gets fixed the proper way.

Moving to nsenterprise+, adding nsBranch. I'm assuming the 0.9.2 branch fix hasn't been checked into the trunk as it wasn't the "complete, right fix."

not happening for 0.9.4. some underlying work Rick is doing will get us closer to this, but we're not there yet.

Checkpoint for nsbranch . . . Judson - Are we there yet on this one?

Minusing this one per my email exchange with Judson. This looks like a good one to get, but it is not gonna be fixed until later in TM0.9.5.

marking nsenterprise-.

*** Bug 96159 has been marked as a duplicate of this bug. ***

-> 0.9.6

->0.9.7 bug #94205 will be landing soon - hope :-)

*** Bug 106931 has been marked as a duplicate of this bug. ***

Is there something corrupt in Comment #158 (from 7-20-01) that causes it not to wrap in the Mozilla 0.9.6 window? I don't know if this is a regression in Mozilla (I doubt it) or something getting messed up in the buzilla database. Also, I apologize to anyone who thinks this is trivial; it is, but I'm hoping it's easily fixed.

that was a bugzilla bug that was recently fixed i believe.

OK. To summarize the current situation and get the ball rolling again.... At the moment we can: 1) Get the right docshell 2) Get the postdata associated with the page 3) Call loadURI with this info (bug 94025 has been fixed). This is not sufficient for doing view source completely correctly, since there could be multiple entries in the cache all with the same URI and same postdata but different content. It _is_ sufficient for fixing bug 64100. Rick said: > there is a cache key per URI loaded for the document. This means that to be > "correct" someone needs to hold onto *all* of the cache keys - including the > ones for linked CSS and JS... This is true in general. In reality, however, we're dealing with the following operations: A) save/save as. This uses nsIWebBrowserPersist now anyway, so I am not sure this discussion applies anymore... At the very least, this should be spun off into a separate bug on nsIWebBrowserPersist if it's still a problem. (bug 84106 should be retested). Using the PERSIST_FLAGS_FROM_CACHE flag should have the desired effect here. B) Send page. Is this going to send all the linked materials together with the page? (I sicerely hope this is not the case; munging urls in the source to point them to the online content seems like a better solution, a la nsIWebBrowserPersist). If it is _not_ sending the linked stuff, then we don't need a "whole page" solution. C) View (frame) source. This could not care less about things linked from the page/frame. All it wants is the source for the page/frame itself as it came off the wire. D) Changing character set. This could actually benefit from the more gneral approach. This seems to cover all the things mentioned in the bugs this blocks. I propose: I) Bug 64100 should get reopened and fixed (this is not very difficult to do now, with the infrastructure we have in place). II) Rick and company should decide whether we actually need a generic api for this (especially since intl seems to be opposed to the concept of fixing item D). I'm getting very tempted to move view source into C++, from whence I could just call nsIDocShell::LoadURI directly and not have all these problems. The current apis would allow this to work just fine.

*** Bug 118487 has been marked as a duplicate of this bug. ***

What about the milestone setting? 0.9.7 is long gone, are we going to have this for 0.9.9 or 1.0??

*** Bug 55583 has been marked as a duplicate of this bug. ***

cc: self

I noted an interesting thing with the current implementation of view source. It does not send cookies to the server? I use cookies for session control. If you try to access the page without cookie you get redirected to login page. When I tried view source I got a blank page? This should only be the case when you open without the session cookie? I dont know if this is relevant or has any impact but I did not se any reference to this situation. I do not have an test case at this moment but I can supply one if the need should arise.

Technically View Source should not be sending anything as is its dependency on this bug. View Source should only show what was received by Mozilla from the server. It should not cause a second request.

This might be redundant information, but I just can't pull myself to read all the post. But it seem to me Internet explorer make heavy use of disk cache. Efficient at it even. When you view source and look at the file, its not a file in meory, but rather a a physical copy. So can't we just stream the source to memory AND disk when we loading a page, and view source just open the file at that particular time. Correct me if I'm wrong, but Internet explorer's view source doesn't apply to dynamically generated content. (So its not fetched from memory, but rather disk) Someone know how opera handle this?

Mike: it is true that IE like mozilla will reuse content from the [disk or memory] cache when it is available. but, unlike mozilla, IE will generate HTML when the [disk or memory] cache does not contain the requested document. This bug is mostly solved already. Mozilla now puts every downloaded document into the cache, regardless of cache control headers (the headers are honored when fetching the documents from the cache for normal page loads). This does NOT guarantee that the document will exist in the cache when the user views or saves the source of a document in a random browser window, but it does pretty much guarantee that it'll work just after the document is loaded. NOTE: i'm only talking about HTTP and HTTPS, not FILE or FTP. though, i think FTP could be modified to use the cache for downloaded material in a similar fashion.

Is http://bugzilla.mozilla.org/show_bug.cgi?id=120809 related? An image gets refeched from the server every time, but definitely is in the cache.

darin: If I go to a page that is generated on the fly (e.g. slashdot), open it in two windows (thus causing each window to have different content) and then save both of these windows, will the saved copies be different?

you will save the most recently fetched version of slashdot. is this a problem we want to solve? if so, then it means pinning documents in the cache, which sort of goes against the idea of having a fixed sized cache. i'm not saying it's impossible... indeed, the cache provides facilities for pinning an entry in the cache, but it certainly is not without its tradeoffs.

My opinion is that "yes, it is a problem we want to solve". I've got some CGI scripts which generate oodles completely different types of content under various circumstances. While debugging, I'll often look at the sources of different outputs (using some non-Mozilla browser). perhaps Yet Another Pref? "Save raw HTML source in cache to avoid new requests when using the view-source command." Turned off by default, easy for web developer types to turn on. Only folks who want the feature have to deal with the tradeoff. -matt

Yes, it is the problem we want to solve, but No, it should not be a pref. This is what you would expect to happen, that the page you look at gets saved. If we don't pin it there is no way to be sure that that happens. IMHO we also want to pin it for at least three other things; view-source, send- page and history. Pinning the htmlsource should not be that expensive, how many html-pages does a "defaultinstall doing normal browsing" contain approx? I have 171 files in my cache, though i guess some of them are images, i would happily waste 10-20 of that on pinned down pages

I might be naive, but when I say "save page as" or "send page", I mean "do it with the stuff I am seeing on my screen right now", and it is very disturbing for me to see a refetch when I do it. Many pages are relatively small and static, most servers are very responsive and many people have good connections, but this is not always the case, so IMHO, treating a refetch as a Very Expensive Operation and doing it only when absolutely necessary or when explicitly asked for is The Right Thing.

As I understand it, we _do_ want to avoid having another refresh. Unfortunately, that _doesn't_ necessarily mean "the contents on your screen right now." Why? Because you can change the stuff on your screen. Fill in forms, execute javascript, whatever. Things of that nature should, IMHO, not be reflected in "view source," "save page as" or "send page". But what about changing the character set? (that's bug 17889 btw) Changing the charset shouldn't cause a page to reset. If we do want to support a save/send page option which maintains the current state of the page, it should be considered a different feature and not interfere with the current effort to implement the current features.

I don't think that saving the page EXACTLY like it looks now is what anyone wants. The ideal situation would be to save the HTML document to the cache as the server gave it to you, and then also save all the other files the server sent over that the document needs to render correctly, such as StyleSheets, images, Java and other ebedded things (Flash, etc) That way, whenever you want to either see the source, or save the entire page (HTML+dependencies) etc it is all already there, moz would just need to copy it elsewhere. The only reason that moz needs to refetch anything from the server is if the user hits Reload or the cache TTL expires. Or if the meta refresh is set, of course.

-- #197 From Darin Fisher 2002-01-27 16:14 said: ------- > you will save the most recently fetched version of slashdot. is this a problem > we want to solve? YES. View Source, Save As, Print -- they should ALL act on the presently viewed page. In the case of view source, we should always view the html for the page that is displayed in the window. If this means that I have 10 browser windows open, all which requested some http://something/time.cgi URL (which outputs the time of day in html) at slightly different times, I should get 10 different view sources. Always. This is vital to being able to develop content for Mozilla, because all major server-side development these days is dynamic, and it is *categorically impossible* to test or debug dynamic content with Mozilla if it does a refetch when the user hits "view source." The same goes for Save As and Print from a user's perspective. If I just hit the "submit" button to place an order, I don't want that posted again if I want to print the invoice page or save it to disk (which would place another order). And if I choose to place two different orders to the same site in two different windows (sequentially), and then save/print each invoice after each has been placed, it should "do the right thing" and save/print two different invoices. ------- #201 From Nick Lewycky 2002-01-27 18:14 said: ------- > As I understand it, we _do_ want to avoid having another refresh. > Unfortunately, > that _doesn't_ necessarily mean "the contents on your screen right now." > > Why? Because you can change the stuff on your screen. Fill in forms, execute > javascript, whatever. Things of that nature should, IMHO, not be reflected in > "view source," "save page as" or "send page". But what about changing the > character set? (that's bug 17889 btw) Changing the charset shouldn't cause a > page to reset. Filling in forms doesn't change the source html. Executing javascript, in most cases, doesn't change the source html either (document.write() is the only exception I can think of, and I believe that is a microsftism), so the browser should just show the original html as sent by the server for save/view source. But this does bring up an interesting print problem... Printing should, IMO, show the current form contents, as well as any dhtml changes via javascript, etc. That information is not contained in the source html.

From previous discussion, I understood that Print worked with the DOM representation of the page anyway. So it's probably Ok already. Who wants to test printing a filled form?

Hey, what's wrong with the formatting of my previous comment? Is it possible to edit a comment after posting it?

> Print Not relevant. Printing prints the DOM, not the source. So you always print exactly what you see. All the concrete examples people have given so far of cases where they would want to see separate source for the same url involve post data that was sent to the URL. In those cases, we _do_ cache each response separately.

The view source problem isn't limited to post data sent to the URL. A "timeofday.cgi" is a good example of a non-post request sent to a URL which returns dynamic content, and the "view source" should be different for each browser window/history/tab. This isn't a contrived example. JSPs work this way as well. As do ASPs. As do PHPs. etc. URL accesses may cause the server to generate and send dynamic content /without/ sending state information via (get/post/cookies/URLrewrite) which must be cached for view source.

(I got a midair collision with somebody saying the same thing as me; this is in response to comment #206) Not at all; it applies to *any* site whose content might change over time. If I visit slashdot.org at lunchtime, leave the window open, do some other browsing in other windows, including visiting slashdot.org again later with new stories, then go to view source of the original page, I expect to get the source of that page, not the source of slashdot.org at some arbitrary later time that I happened to view it. From a user's perspective, this is basic functionality. If the page is sitting right there in a window, why should it be impossible to view its source? I do understand the implementation reasons why it's hard, but it's also hard to implement a functional CSS engine and HTML4 viewer - that's not a reason to choose not to do it, or we'd still be using MozillaClassic :) It should *never* be possible to have a page in a window and not be able to view source, or send or save the page, and get any result other than the exact content of the window that is on your screen. Even if the page specifies no-store. That's what users expect; it should be what we do. No exceptions.

i think one point needs to be made clear: it is trivial (modulo some API issues) to fix mozilla to not hit the server for view->source and file->saveas nearly 99% of the time (including POST results). it is far more complex to ensure that this is correct 100% of the time, and it is also far more complex to ensure that view->source and file->saveas correspond to the page being viewed. of course, i understand why solving this is important, and i would love to see it solved. i also think that we have to be careful when we go to solve this problem, since there is a bloat factor to consider... even if it wouldn't be an issue for the majority of users, we still need to think about people with systems "circa" 266 Mhz. moreover, some users disable their disk cache. we need a solution that works (or can be disabled) when there is no disk cache.

Another thing I wanted to mention is "back" (the leftmost button in the toolbar). moz treats "back" as a _refetch_ (unless it's an anchor in this page). IMO this is wrong: "back" means "give me what I already saw", not "refetch". Again, treating a refetch as a Very Expensive Operation and doing it only when absolutely necessary or when explicitly asked for is The Right Thing.

sam: back currently loads the page from the cache. it is true that if that page is dynamically generated and has been more recently visited and hence modified that you will not see what you originally saw. fixing this would be difficult and probably should not be done as it would result in some serious bloat, especially for SSL pages which cannot be cached on disk for security reasons.

Does the cache not have a (timestamp visited), (timestamp last modified) for the entry?

yes it does, but how would that help this situation? sure it would allow you to determine that the page in the cache is not the same page as you previously visited, but then what? you'd still be without the original page, so why not just show what you've got in the cache.

What your looking as it the need to cache everything but (invalidate/show/hide) the cache on specific requests (or specific components.) It's agreed that Print, Save As, View Source and some other commands do not need to reload the URL from the network. One option to alleviate this, is to cache all requests in both digested and raw formats regardless, but make them visible only to specific services (Print, Save As, View Source, etc.) The "stealth cached" items expire and are replaced/expired as a normally cached item (perhaps with a rule always cache). By using this method, you can have one efficient cache for the system. * Cache everything * On cache request, determine visibility to decide if you should return item is not in cache (even if it is) and must be reloaded from the network. ** If do not cache pref is set, then the item isn’t visible in the cache (to the browser) and must be refetched. ** If Print, Save As, View Source, all cache is visible always. This would involve some checking between components to see who is authorized to view the stealth cache, but I think it would improve performance in addition to eliminating double posts and gets. These are just suggestions and probably nothing new, but they might help in the discussion for a direction to go...

thomas: most of what you've described, mozilla already does (modulo some bugs). it achieves the effect using load flags. there are three principle load flags: LOAD_NORMAL, LOAD_FROM_CACHE, LOAD_BYPASS_CACHE. clicking on a link or entering an address in the URL bar sets the LOAD_NORMAL flag. file->saveas, view->source, file->print set LOAD_FROM_CACHE, and shift-reload sets LOAD_BYPASS_CACHE. i should really say that this is how mozilla is designed... there are bugs and mozilla may not set the load flags correctly in all cases. what mozilla does not do is "pin" entries in the cache. this means that a page you are viewing could be evicted from the cache if you spend sufficient time browsing in another window. as a result, file->saveas, etc. may require a server hit.

if(cached(current_page)) { &nbsp;&nbsp;&nbsp;display_source(cache(current_page)); } else { &nbsp;&nbsp;&nbsp;echo "Sorry, the page in question was not found in the cache, click here to re-retrieve this page"; } If the a refetch is required, the user should be well aware of it (and mozilla should ask permission), period. Else I assume that's the source of the current page. This is just my take/solution on the whole thing, and what I expect should be done, obviously the real code will probably be quite a bit more complex than my lame psuedo code. :) Is it really that unacceptable to say "sorry I couldn't view the source, here's why:"?

steve: you have described exactly what mozilla already tries to do (again, modulo some bugs).

i think that a seperate bug should be filed to lock a window's history in the disk cache, but i feel that full source versions of pages should not be stored in memory. if someone decides to turn off disk cache it should be acceptable to require a refetch from the server on view source. what people are attempting to add into this bug would be best implemented by changing the cache system--anything else is just a hack. i don't think anyone wants to see the cache system extended this close to 1.0. file a bug and dep it on this one. i see wins here if the cache is modified to provide for a true history, especialy since the cache's traditional role is all but obsolite due to dynamic pages and fast connections. this really should be discussed in a news group.

darin@netscape.com 2002-01-30 11:54 wrote (#215): > what mozilla does not do is "pin" entries in the > cache. this means that a page > you are viewing could be evicted from the cache if > you spend sufficient time > browsing in another window. as a result, > file->saveas, etc. may require a > server hit. The only problem with this, and I don't mean to harp on this point if it is already abundantly clear, is that Mozilla's DOM is effectively a "memory cache", and it does get out of sync with the real cache. If I can see the page in the mozilla window, it is only reasonable to expect that I can view its source, save it to disk, or send it in email, WITHOUT requiring the browser to get it from the server again. My humble picks from the best solutions yet suggested: 1) Sync the caches. Sync the caches. Sync the caches. Either (a) purge DOM from memory when a page is purged from disk, or (b) pin entries in the cache. (a) is the same fix as popping up a "page is gone" requestor, but doesn't leave the user scratching his head wondering why in the world Netscape's engineers can't save the page to disk if they can render it right there on the screen, because it will no longer render to screen. If it's gone from the cache, it can't be rendered without a request. or 2) add yet another cache that mirrors the DOM, saving the pure HTML. Better yet, attach it somewhere as part of the DOM. This eliminates the sync problem because if Mozilla every goes to the cache, it will reconstruct the DOM from (and with) the html. IMO, 1 is the best "fix it now!" solution and 2 is the best "real" fix, even though it is either ugly or less memory efficient. If I'm speaking out of my ****, forgive me. Trying to help.

> purge DOM from memory when a page is purged from disk You mean blank out the page the user is currently viewing? The only place the DOM is "cached" is in the actual view the user has. As soon as you view something else the DOM is destroyed.... > If it's gone from the cache, it can't be rendered without a request. Correct. The if you hit "back" to go back to a page that's gone from the cache you will get a request (and an alert). The problem is when the source has already been rendered.

OK. And then we can come to conclusion that the only correct solution is to hold a direct cache key to pinned cache entry for every dom which is held in mozilla. But this was already suggested. If i try to get back in history to other page and it's cached I load it and then I get a new direct cache key. So there is no need to mess up with direct cache keys and history. Certainly this behaviour is not perfect but it can't be without other tradeoffs. Note that there are other bugs where some people want to have cached the entire DOM for purpose of page history. Then in this DOM history should be cached the cache keys for sources.

Sorry for the unconstructive spam but... If the only way to fix this is to modify the cache to allow "pinning" pages in the cache, we've had over 1.5 years to realize this and so I *do* think Mozilla 1.0 should wait on it. Also, at this point there have been SO MANY examples and counter-examples that I don't remember what works and what doesn't. The basic idea seems to be: the original HTML for whatever is displayed (in a Mozilla window) should be accessible to various functions (Print, Save As, View Source, etc.) without a refetch. Is this not correct, i.e. do these functions already work as I just described?

It seems that we are down to 2 areas of misbehavior: 1. GET requests that have been evicted from the cache will hit the server *without* asking the user first. 2. POST requests do not work. To address these problems i'll expose some kind of nsIWebPageDescriptor interface that allows access to the following information: a) URI b) PostData c) Cache key There will also be a LoadURI(nsIWebPageDescriptor) -- somewhere... Additionally, i'll make sure that if the 'load-from-cache-only' load flag is set on the view-source channel (which it should be) then the user is prompted BEFORE a request is made to the server (in the case where the page has been evicted from the cache...) This is basically what boris proposed way back when in comment #186 ;-) -- rick

If the majority of dynamic pages contain a "no-cache" meta tag, does the "cache-pinning" solution mean that we're back to square 1 for developers wishing to see their outputted page source, and for anyone wishing to save-as or send a dynamic page (e.g., an invoice)?

Question: "Pinning" refers to preventing the expiration from cache of any element currently displayed in a browser window? If not, what exactly does it mean?

Al: 'no-cache' documents are still put into the cache. they are there for the purposes of view->source, etc. normal page loads of a 'no-cache' document will bypass the cache.

How about no-store documents? Is there any way to ensure that those get pinned, say, only in the memory cache - so that they can still have view-source done on them, but will never ever enter the disk cache?

stuart: yes, even 'no-store' documents are cached (in the memory cache only) for the purposes of view->source, etc. this was only recently changed to be this way... previously 'no-store' documents were not cached.

*** Bug 120457 has been marked as a duplicate of this bug. ***

darin: Would it be possible to associate a particular cache entry with each document currently on the screen, and (through the magic of reference counting or something along those lines) guarentee that that cache entry not be overwritten so long as the relevant page is open? That would mean that the cache could end up containing three or four copies of a page with the same URI, so that view source would work (you would use these cache entries to load the page in view source or save as); as soon as those windows got closed, the entries in the cache would be deleted (or marked as out of date or whatever). In the real world this would not cause much bloat (how often do you visit the same page in multiple windows?) but it would solve the view-source-of-the-same-page-multiple- times problem.

hixie: the cache already provides support for exactly what you're describing. we'd just need to hook up docshell and friends to utilize it. there are issues, of course, that need to be resolved... consider the hypothetical example of an ever increasing number of windows (eg. you've just visited an evil website)... at what point does the cache refuse to hold items in the cache? does it ever refuse? does it honor or ignore its size limits? it seems to me like we need a solution that allows the cache to honor its size limits, otherwise we run the risk of serious meltdown resulting from a malicious website.

Also, the cache would only hold multiple entries for the same URI if HTTP thought they resulted in different documents. Otherwise, the reference held by each window points to the same cache entry.