[boost] Is there interest in a library for object (especially STL object) marshalling?

Subject: [boost] Is there interest in a library for object (especially STL object) marshalling?
From: Roger Sanders (roger.sanders_at_[hidden])
Date: 2013-10-09 22:07:51

Hi there,

I'm not totally sure if this is the right "fit" for boost or not. Perhaps
this problem isn't general or abstract enough to warrant inclusion within
boost, but I thought I'd send an email around to see what people thought.

I've written a library that I've been using extensively, both for my own
code, and more recently for a large project in my work, that allows STL
objects to easily be exchanged over a function call, even if the caller and
the target function represent those STL objects using incompatible binary
layouts. This is intended in particular to deal with the common problem of
sharing STL objects between multiple compiled assemblies, such as a main
executable and satellite DLL's. I'm already using this library extensively
to exchange STL objects over pure virtual interfaces for a plugin-based
application.

The traditional advice people give is "don't use STL objects across DLL
boundaries", but this is impractical for applications that are heavily
modularized, or plugin-based applications with complex interfaces. Dealing
with raw arrays and pointers is often difficult and error-prone, hence why
we have STL containers to begin with. Returning data from a function call
can also become extremely difficult when STL containers aren't used. With
an array for example, if the total number of elements cannot be known by
the caller, this can lead to fixed-sized buffers being used with hard-coded
maximums, or functions requiring multiple calls, either to determine the
size of the data, or to free an allocated buffer that's been returned.
Attempting to use STL containers across assembly boundaries without
addressing STL compatibility issues however basically requires you to use
the exact same compiler version and compiler flags when building each
assembly, which is not really possible with a plugin system designed to
allow third-party plugins, or even simply with assemblies built at
different times on different systems, and it's also inconvenient for
development, since you can't easily compile just one assembly with debug
compiler settings, while leaving the other assemblies compiled with release
compiler settings.

Ideally, you want the same signature for a function or method, whether the
target code is contained within the same assembly or not. The goal is to
make these assembly boundaries "invisible", so you can just write the same
code to access functions, regardless of where the code for the target
function is located. With the help of this library, along with a simple
inline wrapper function which the caller is basically unaware of, that can
be achieved, and existing code which makes use of STL objects can be
shifted into separate assemblies without refactoring. I haven't found
another library like this in the public domain, that attempts to solve this
issue comprehensively for any type.

Some of the points about my current library implementation are as follows:
-Fully template based, with all code defined in header files.
-Support for all *primary (see exclusions below) STL container types up to
C++11, including strings and keyed containers.
-Supports infinite nesting of STL container types, IE:
std::vector<std::list<std::map<std::wstring, std::stack<unsigned int>>>>
-Heavily optimized, with profiled performance comparable to the best
possible hand-written marshalling code.
-Extensible, so new STL container types, or boost container types for
example, and potentially user-defined types, can be supported and
marshalled in a consistent manner, even when nested within each other.
*The only three containers I'm aware of that I haven't supported yet are
std::pair, std::tuple, and std::bitset, although I can include support for
them easily enough. Although difficult, I can also think of how to add
support for pointer types such as std::shared_ptr, where object lifetime
and destruction can be correctly and automatically managed across
assemblies when these types are marshalled.

The only compiler requirement here is that each assembly uses a compatible
C++ ABI, or at least, as much as is required to allow calling a virtual
member function on an object that was constructed within another assembly.

What I wanted to get an idea on is whether there is any interest in having
something like this included as part of boost? Perhaps there's enough scope
here for a general "marshalling" library, that would address some other
similar concerns in the future? There would be a fair amount of work to do
in order to "boostify" the library and write the corresponding
documentation, but I'd be willing to do it if there was interest. Any
feedback, thoughts, criticisms?

Thanks,
Roger Sanders.