Re: [boost] Is there interest in a library for object (especially STL object) marshalling?

Subject: Re: [boost] Is there interest in a library for object (especially STL object) marshalling?
From: Roger Sanders (roger.sanders_at_[hidden])
Date: 2013-10-16 01:36:29

On 11 October 2013 09:30, Gavin Lambert <gavinl_at_[hidden]> wrote:

> On 10/10/2013 3:07 PM, Quoth Roger Sanders:
>
>  The only compiler requirement here is that each assembly uses a compatible
>> C++ ABI, or at least, as much as is required to allow calling a virtual
>> member function on an object that was constructed within another assembly.
>>
>
> That could be a steep requirement, since AFAIK vtable location and layout
> isn't standardised, especially once you start getting into multiple and
> virtual inheritance.  And name mangling can get in the way of finding the
> things to be called in the first place.
>

I'd love to have something that didn't rely on ABI compatibility, but I
don't think that's really going to be possible, or at least, not while
adhering to the portability requirements of a boost library. With the new
standard layout type definition in C++11, and with the help of the new
alignment querying/control in C++11, it's now finally possible to produce a
data structure that you can create in memory and guarantee you can
correctly read in code compiled under a different compiler, without relying
on undefined or implementation-specific behaviour. Unfortunately, while
sharing data is nice, C++11 has done nothing to help with the bigger issue
of sharing code. Using the extern keyword with a linkage specification (IE,
as in the extern "C" usage) is too limited and tries to combine a few
different concepts together, and as a result, ends up doing a poor job at
all of them. What we really need is a new keyword, separate to the extern
keyword, that is specifically intended to specify a platform-specific
function "format", IE, the calling convention, that the compiler should use
for that function. In particular, as opposed to the extern keyword, it
should:
1. Be able to be applied on member functions as well as non-member functions
2. Be able to be used on overloaded functions
3. Not affect function linkage when used (IE, doesn't mark a function as
having external linkage)
With a new keyword that meets this requirement, you'd be able to actually
mark a given function as explicitly having a particular calling convention.
As long as two compilers support a compatible calling convention, you could
then safely invoke that function from code compiled using another compiler,
or even another language. The calling convention is still "implementation
specific", as it has to be, since calling conventions are by their very
nature platform specific, but it's now supported by the language, with
compile-time checking for compatibility. With support for calling an
individual function with compile-time checking for calling convention
compatibility, you could then just provide a base class which meets the
requirements of a standard layout type, and contains function pointers to
each exposed function, with some inline wrapper functions to invoke them.
This would give identical usage syntax to calling the methods natively,
fully checked for compatibility at compile time, with behaviour guaranteed
by the C++ standard, without requiring ABI compatibility.

Without this kind of an enhancement to the language itself, I don't think
it's really going to be a satisfying result attempting to provide any kind
of boost interop library that attempts to provide interop without assuming
some level of ABI compatibility. I am optimistic on another front though.
The simple fact is, for any given platform, there's a limited number of C++
compilers people actually use, and most of them attempt a degree of
compatibility anyway. The biggest problem is really on x86/x64 systems,
with GCC and MSVC being incompatible. Clang is rapidly developing though,
and I've seen there's a push for Clang to have full ABI support for both
GCC and MSVC, IE, as a compiler switch to select between the two. Once we
reach that point, it really just becomes a matter of any given software
system selecting a base ABI to use, and Clang can then be used regardless
of that choice to compile compatible code for that system. With this kind
of solution in place, you could then just write code relying on ABI
compatibility, and implement a language-supported solution when one becomes
available (IE, just by marking all functions on an exposed interface with
the kind of keyword I've proposed above).

Whatever you attempt with ABI compatibility though, you still need to solve
the STL issue as a higher-level problem though, which is what I'm proposing
to add to boost at this stage.

> And memory management is always entertaining given that all sorts of weird
> and different allocators can be in use even before you introduce a
> different compiler into the mix.  (Though this is where shared_ptr and
> unique_ptr's pointer+deleter concept can get you out of a jam, although
> it's more common to use opaque handles and explicit destroy functions.)
>

I may be able to package up something for boost on the memory management
side that can help too actually. In my work, we had the issue of wanting to
be able to create an instance of a type, where only a base interface of
that type was known to external code, with the actual implementation being
completely internal to a particular module. We wanted to be able to easily
create these types without having to specifically call allocators and
deallocators (especially the deallocators, to avoid memory leaks), and we
needed to be able to pass these types back from function calls, again,
without the caller having to manually deallocate them. Something like
shared_ptr was unsuitable though, not only because it felt too heavy-handed
(we didn't need or want the reference counting), but also because it
couldn't be passed safely across DLL boundaries either. My solution was a
very thin "pointer" type, which basically just referenced the allocator and
deallocator within its constructor and destructor, and used C++11 move
semantics to allow returning them easily from function calls. The final
version was macro-based to automatically generate these pointer types for
any interface type, with only a couple of macro calls to create all the
necessary code, including the allocators and deallocators. This pointer
system, along with the STL interop code, are the basis for our
communication model we use between components, and the payoff is very
significant. We have easy bi-directional communication across pure virtual
interfaces, with STL objects being shared natively on those interfaces,
with minimal overhead and without ever having to manually call an allocator
or deallocator, or manually unpack or repack an STL container. You can
write code which creates a type and calls functions on that type, which is
actually calling an allocator from a different module and invoking
functions over a pure virtual interface for that type, with STL objects
being shared, that looks identical to creating that type on the stack and
calling functions on that type as if it was defined within the same module,
with a compatible STL implementation. I could have another look at this
code and see if it could be improved. I think it should be possible to
build a more general fully template-based solution that doesn't rely on
macros, in which case it might be a worthwhile addition along with the STL
marshalling classes.