Re: [boost] [RPC] Compile-time reflection for RPC and mocking

Subject: Re: [boost] [RPC] Compile-time reflection for RPC and mocking
From: Thomas Heller (thom.heller_at_[hidden])
Date: 2014-12-09 12:18:48

Hey there,

On Monday, December 08, 2014 15:27:33 Peter Bindels wrote:
> Hi all,
> 
> I've searched through the Boost mailing list history to find related emails
> and the last I found was from January where Oleg Labutin and Hartmut Kaiser
> discussed compile-time reflection in the context of RPC mechanisms.
> 
> Based on Hartmut's speech last Saturday introducing HPX and explaining a
> bit about async/future/promise I tried to implement a simple RPC mechanism
> that uses then to decouple work and the result I have so far is very
> promising. I have a simple interface with a call that returns a future, and
> the caller does not distinguish between calling the actual object and
> calling on a proxy, as both return a future<T>.

(This is one of the HPX developers speaking)
Please bear with me, I didn't attend meetingcpp and don't know which questions 
have been answered already or not. I just would like to add my two cents here.

The biggest question I currently have here is: How do you distinguish if it is 
remote or local?
The HPX answer to that problem is a distributed database which is responsible 
for mapping Global Identifiers (GIDs) to actual objects. We call that the 
Active Global Address Service (AGAS). The GID therefor acts as your proxy 
which is, in HPX, kinda like a void pointer to a possibly remote object where 
you can invoke so called actions on. Actions our mechanisms to do RPC, more on 
that below...

By meaning "return a future<T>", do you refer to std::future<T> or your own 
class behaving like a future<T>?

> It does what the discussion
> mentions - serialize the name of the interface and function, as well as the
> arguments (basically the mangled name of the interface function) to ensure
> there are no mismatches.

You say "basically the mangled name", what does that mean exactly? How do you 
get unique names?

> 
> I would like to take this oppurtunity to propose two things:
> 
> - I can work on this idea to work it out into a full RPC mechanism, to be
> proposed for Boost.RPC, if there is enough interest. Currently it uses
> macros to generate and wrap functions, and a simple serializer that
> serializes in the same way that Protobuf and Thrift also serialize.

As has been noted in a different mail in that thread, boost already has a 
solution for that: Boost.Serialization. We currently use it but were not very 
happy with its performance (and still aren't), but that's a different issue ...
I think the interface that Boost.Serialization exports is great to implement 
compile time reflection without compiler support. It has its caveats, but it is 
a decent, and accepted way of accomplishing this task.

> - Given a compile-time reflection method, the entire proxy class and
> dispatch class can be auto-generated from the interface class - the full
> code for registering an interface with the RPC mechanism would collapse to
> RPC::Register<T>(); To do so, it would need two language additions. It
> needs access to a compile-time iterator over functions and traits about the
> argument types, return type, name and whether it is a virtual function. For
> the prototype this can be manually generated for an interface to show it
> works. Additionally, it would need to be able to use such a definition to
> instantiate a virtual function from a base class - basically, hooking up an
> std::function<X> to a virtual function X directly. I think this is
> well-implementable (probably reusing most code from lambdas for the link
> code), but I don't know whether that's true.

Mechanisms for that are already there: Boost.Serialization. Nevertheless I'd 
be very interested to see the code of your prototype.
Here is more on actions: In order to turn a regular function into a "RPC"able 
action, you essentially need a way for the communicating partners which 
function should get executed on the other hand. The simplest way to do that is 
by it's function pointer and the arguments that are supposed to be passed to 
that function. The function pointer, and it's type is the hard part here. We 
currently solve this by having a template (instantiated on all communicating 
partners) which is serialized over a polymorphic base pointer. As such you 
only need to register this function as a action with your serialization 
library and call it a day. Side note: We haven't found a truly portable way to 
automate that, in addition, with this techniques, we are not able to send 
lambdas over the wire, and the committee had come to the conclusion that it 
isn't possible with the current standard, IIUC.

<snip>

> Any ideas? Reactions on RPC or Mocking?

I like the idea of having a generic RPC library lying around somewhere.
The requirements, i can think of right now, for us in order to use it would 
be:
1) Extension mechanism for the actual network communication
2) Extension mechanism for the actual execution on the remote end
3) Extension mechanism for the future which gets returned
4) High performance of the serialization with zero copy support (preferably 
API compatible with Boost.Serialization)

With that being said, HPX is a little more than just a library for doing RPC, 
there is a whole bunch of stuff we implemented to make it convenient and fast 
to use (and we still aren't really there yet ...). It would be great though if 
we could out source some of our infrastructure to 3rd parties to ease our 
maintenance burden ;)

> 
> Best regards,
> Peter Bindels
> 
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost