I'm a bit new to boost and am trying to determine if I can coax the existing boost serialization system to work for a fairly standard asset loading system. Assets in my domain of game programming are things such as bitmaps, model geometry, audio data, shaders, and various chunks of meta data attached to them. The quantity of such data should always be assumed to vastly exceed the ability to load it all at once, which means getting sneaky when dealing with persistence. From my point of view the boost::serialization code is missing a lot of important features, so I am kind of surprised that they are not there.
A) Serializing for persistence: Assets can reference other assets via pointers, but serializing for persistence should only store names of other assets, and query a resource manager to help convert the pointer into a name (or vice versa). A model object might reference via a pointer a shader and a texture in addition to its geometric data. When I serialize the model for persistence I want to store the geometric data, but when it comes to the shader and textures I only want to store their names. I could manually do this in each method's serialize methods. However the serialization system should also be able to do this via type traits, so that I don't have to hack up the serialize methods. The type trait method for this is preferable, since the method bodies of serialize could then be written as the originally would want to be, so that completely different kinds of archivers can operate on them, especially ones that depend on actually being able to walk into the pointers. Its the archive's call how to walk the data really.
B) Serializing for garbage collection: A full set of raw assets for an engine can easily push into the terabyte range, which means you never actually work with all of the data at once. To work you need to be able to load what you need as you go, but to keep your working set down you need to evict unused assets as much as possible. Another type trait could be used to build a list of reachable objects containing the type trait from an array of root objects which are serialized. You can then compare this list to the full list of what the resource manager has currently loaded, and either delete or save the modified but now unreachable nodes. This system has the advantage of being useful for other purposes, such as making making an asset browser, without having to also implement a full blown system for reflection and introspection.
C) Serializing for counting memory: You can also make the archive simply count the memory overhead of your objects. It needs the ability to count memory without counting the memory twice, and it also needs to be able to either do this for a single object (
i.e. don't follow reference to other assets), or for a hierarchy (i.e. 'how much memory does using this asset require, factoring in its dependencies').
So my real question boils down to : I can't really be the first person to want it to be able to do these things right? Am I crazy for wanting to coax the boost serialization code into working this way?