Re: [boost] [serialize] Handling multiple files, deferred marshalling, and progressive loading

From: SeskaPeel (seskapeel_at_[hidden])
Date: 2006-07-04 07:46:29

Hi Robert,

> > 1/ First, we are using multiple files, which can be created at
> > different moments. The main reason is that we want to be able to
> > share resources contained in one file for multiple projects. An alarm
> > rung in my head
> > telling me that serialize won't be able to handle this case,
>
> why not?

Because I saw nowhere in the docs how serialize would restore pointers when
loading from multiple files. Suppose I load file1 that contains a resource
named "r1". After file1 is fully loaded, and say 5 minutes later, I load
file2. Inside that file, there is a resource "r2" that needs a link to "r1",
how will this case be handled?
I suppose I'll have to manually iterate over the freshly loaded resources
and check if they need to be "post-loading associated". As I can't know if
these resources need or not this last step, I'll have to check each time I
load a file, and thus, I could handle the internal linking as well in this
step, though it will be easier (and more efficient) if it's done by the
loading lib.
So, is there something I misunderstood, or some feature I overlooked?

> > 2/ And secondly, we want to be able to load files progressively, say
> > 100KB
> > by 100KB. Once the file is completely loaded, the pointer restoration
> > can happen. Does serialize support such feature or plan to? If this
> > is not a
> > huge work, is there a way to provide help to get this feature quickly?
>
> I'm not sure what this means.  Taking a wild guess, I would seem that want

> you might want to do is run serialization on a separate thread.  If you
> want 
> to "interrupt or suspend that thread, this would require support from a 
> custom archive.  Such a custom archive would be derived from one of the 
> existing ones.

Not necessarily from a thread, but the aim is to suspend the loading of a
file, and then resume it some time later. What would be even better would be
to specify how long or how many bytes should be read before the loading
function suspends and returns.
I read about your custom archives (some time ago I have to admit), and it
didn't seem an evidence to me that I could implement this feature. Could you
provide some more hints?

> > Today, we have a manual phase of association that is called after all
> > files are loaded. Each time we load a new file, we "post-load" all
> > the resources contained in it, and pointer restoration is achieved
> > this way.
>
> In the serialization library, pointers are restored "on the fly" (depth 
> first).

Yes, that's why I'm considering porting to it :) What does "depth first"
means?

> Robert Ramey

Cheers,

        SeskaPeel.