[Boost-users] [Serialization] serializing handles to pointers, and serializing as garbage collection

From: Sean Cavanaugh (worksonmymachine_at_[hidden])
Date: 2007-07-05 04:16:20

I'm a bit new to boost and am trying to determine if I can coax the existing
boost serialization system to work for a fairly standard asset loading
system.  Assets in my domain of game programming are things such as bitmaps,
model geometry, audio data, shaders, and various chunks of meta data
attached to them.  The quantity of such data should always be assumed to
vastly exceed the ability to load it all at once, which means getting sneaky
when dealing with persistence.  From my point of view the
boost::serialization code is missing a lot of important features, so I am
kind of surprised that they are not there.

A)  Serializing for persistence: Assets can reference other assets via
pointers, but serializing for persistence should only store names of other
assets, and query a resource manager to help convert the pointer into a name
(or vice versa).  A model object might reference via a pointer a shader and
a texture in addition to its geometric data.  When I serialize the model for
persistence I want to store the geometric data, but when it comes to the
shader and textures I only want to store their names.  I could manually do
this in each method's serialize methods.  However the serialization system
should also be able to do this via type traits, so that I don't have to hack
up the serialize methods.  The type trait method for this is preferable,
since the method bodies of serialize could then be written as the originally
would want to be, so that completely different kinds of archivers can
operate on them, especially ones that depend on actually being able to walk
into the pointers.  Its the archive's call how to walk the data really.

B)  Serializing for garbage collection: A full set of raw assets for an
engine can easily push into the terabyte range, which means you never
actually work with all of the data at once.  To work you need to be able to
load what you need as you go, but to keep your working set down you need to
evict unused assets as much as possible.  Another type trait could be used
to build a list of reachable objects containing the type trait from an array
of root objects which are serialized.  You can then compare this list to the
full list of what the resource manager has currently loaded, and either
delete or save the modified but now unreachable nodes.  This system has the
advantage of being useful for other purposes, such as making making an asset
browser, without having to also implement a full blown system for reflection
and introspection.

C)  Serializing for counting memory: You can also make the archive simply
count the memory overhead of your objects.   It needs the ability to count
memory without counting the memory twice, and it also needs to be able to
either do this for a single object (i.e. don't follow reference to other
assets), or for a hierarchy (i.e. 'how much memory does using this asset
require, factoring in its dependencies').

So my real question boils down to : I can't really be the first person to
want it to be able to do these things right?  Am I crazy for wanting to coax
the boost serialization code into working this way?