Re: [boost] Is there any interest in a library for actor programming?

Subject: Re: [boost] Is there any interest in a library for actor programming?
From: Dominik Charousset (dominik.charousset_at_[hidden])
Date: 2013-05-20 13:12:01

On May 20, 2013, at 3:31 AM, Vicente J. Botet Escriba <vicente.botet_at_[hidden]> wrote:
> Le 20/05/13 01:48, Dominik Charousset a écrit :
>> On May 18, 2013, at 11:54 PM, "Vicente J. Botet Escriba" <vicente.botet_at_[hidden]> wrote:
>> 
>>> Le 17/05/13 23:26, Charousset, Dominik a écrit :
>>>> Hi,
>>>> 
>>>> is there any interest in a library for actor programming?
>>>> 
>>>> The library we've developed is a C++11 open source library named "libcppa". It is currently released under LGPL2, but re-releasing the library under the boost license is not an issue. You'll find all important links and ressources either under http://libcppa.org or on the GitHub project page: https://github.com/Neverlord/libcppa, including a user manual as HTML or PDF. We've also submitted a paper to the C++Now conference.
>>>> 
>>>> What is actor programming? For those of you who aren't yet familiar with the actor model: actors basically represent tasks or services that communicate via message passing. In libcppa, actors are very lightweight (you can spawn literally millions of them) and scheduled cooperatively using a thread pool (although actors are allowed to opt-out of the cooperative scheduling). Some of the main strengths of this programming paradigm are: (1) network transparency, (2) race conditions are avoided by design, (3) a strong failure model that enables developers to build reliable distributed systems, and (4) the actor model addresses both concurrency (multiple actors on one host) and distribution (any number of actors on any number of hosts).
>>>> 
>>>> A message is simply a tuple of values that is pattern matched at the receiver. For example, " on_arg_match >> [](const string& what) {...}" defines a message handler that is called whenever a tuple with one single string element arrives. "on_arg_match" simply defines a pattern that matches the signature of the given lambda.
>>>> 
>>>> Please have a look at the examples folder on GitHub (https://github.com/Neverlord/libcppa/tree/master/examples) or at the user manual (http://neverlord.github.io/libcppa/manual/) if you want to see more examples.
>>>> 
>>> Hi,
>>> 
>>> This is really a lot of good and interesting work on your library.
>> Thanks.
>> 
>> 
>>> Moving all the library to the Boost standards would be quite a lot of
>>> work however. Replacing some classes by the corresponding Boost ones.
>>> 
>>> I would see already several Boost libraries in your libcppa library:
>>> * Atoms
>>> * CowTuples
>>> * Dynamically Typed Tuples - type erased tuples
>>> * Tuples Pattern matching
>>> * Actors
>> I agree. At least separating pattern matching (for dynamically typed tuples) from the actual actor implementation would make sense.
>> 
>> 
>>> There is something that I would like to see on the Actors architecture.
>>> The interface of an actor is not really defined other than by their
>>> behavior. It can receive any dynamic tuple. This is a lot. I would like
>>> a more statically typed interface. An actor could define statically the
>>> kind of messages/signals it is allowed to handle and make it part of the
>>> interface.
>> There was some discussion about this topic after the talk at the C++Now conference. I do agree that it is desirable to enable the compiler to verify the correctness of actor programs at compile time. However, in order to be able to ensure the correctness of a program, one would have to define not only all possible input messages, but the response types depending on the input as well. Consider this example:
>> 
>> actor_type<reacts_to<atom("add"),int,int>::with<int>,
>>            reacts_to<atom("hello")>::with<std::string>> my_actor() {
>>     return (
>>         on(atom("add"), arg_match) >> [](int a, int b) {
>>             return a+b;
>>         },
>>         on(atom("hello")) >> [] {
>>             return "world";
>>         }
>>     );
>> }
>> 
>> It's easy to check whether the interface is fulfilled. On each send, the compiler is able to check whether the given message matches actor_type and if the sender of the message (more precisely, the receiver of the response message) can handle the result. However, an approach like this cannot allow an actor to change its behavior. Hence, something as simple as the classical dining philosophers example (https://github.com/Neverlord/libcppa/blob/master/examples/message_passing/dining_philosophers.cpp) is not possible.
> What do you mean by change its behavior?

To change the behavior of an actor means to replace the current message handler by another one, i.e., call become().

>> So, what if we allow actors to set a subset of given actor_type as behavior? Consider this example:
>> 
>> struct my_actor : actor_type<reacts_to<atom("init">,
>>                              reacts_to<atom("add"),int,int>::with<int>,
>>                              reacts_to<atom("hello")>::with<std::string>> {
>>     void init() {
>>         // check whether given match_expr defines a subset of defined actor_type
>>         become (
>>             on(atom("init")) >> [=] {
>>                 become (
>>                     on(atom("add"), arg_match) >> [=](int a, int b) {
>>                         return a+b;
>>                     },
>>                     on(atom("hello")) >> [=] {
>>                         return "world";
>>                     }
>>                 );
>>             }
>>         );
>>     }
>> }
>> 
>> This approach only allows class-based actors, because we need to evaluate the type information provided by 'this'. We would enable the compiler to check for *some* errors, but we could not verify, at compile time, that an actor actually implements its own interface, because we have no way the check whether an actor implements a message handler for each message it claims to be able to receive.
> Good point. I think that your approach is to let the message on the mail 
> box until a match is requested. On the systems I use to work an 
> asynchronous message is simply destroyed if not handled explicitly, so 
> in my case the message is always implemented implicitly. Another history 
> is for the synchronous communication (reply?). I would say that this is 
> similar to callback interface.

Yes, each message remains in the mailbox until the actor handled it. However, it's easy to define a catch-all rule to discard unexpected messages:

void my_actor() {
  become(
    on(atom("foo")) >> [] { reply(atom("bar")); },
    others() >> [] { cerr << "unexpected message: " << to_string(self->last_dequeued()) << endl;
  );
}

>> 
>> Even worse, both approaches do not allow actors to send/receive messages before replying to a request (unless message passing would be blocking, which would put an end to scalability). If we go back to the reply() API currently found in libcppa, we couldn't match request and response type.
> You lost me.

The compiler needs to know what response type belongs to what request type. The only way to do this in C++ is to return the response from the request handler, so it becomes part of the signature. Using reply() wouldn't work, because we cannot determine from what handler it was called.

>> 
>> In any case, we would no longer be able to converts threads to actors on-the-fly:
> Humm,  this would be unfortunate.
>> int main() {
>>     auto worker = spawn(...);
>>     // Um... what is the type of 'self'? Is it a valid receiver for the response message?
>>     send(worker, ...);
>>     receive (
>>         // again: what is 'self' allowed to receive/reply? ...
>>     );
>> }
> Couldn't spawn have a template parameter giving the provided protocol?
> Self could be typed with the supported protocol(s). Self could be passed 
> by the underlying system as a parameter of the spawned function, as e.g. 
> Boost.Coroutine does. This avoids the use of thread specific data to get 
> the current self as I expect you do. Up to the user to manage with the 
> self parameter.

The 'self' pointer has to use a thread-local variable, because (1) scheduled actors migrate between threads (see Oliver Kowalke's response) and (2) threads are converted implicitly as soon as they use actor-related messages such as 'send'.

>> Long story short: a type-safe interface would be less powerful and would require more boilerplate code. However, perhaps there is a middle ground for doing type checks at runtime when compiled in debug mode?
>> 
>> int my_actor() {
>>     // macro, defined as nothing if not compiled in debug mode
>>     assert_protocol(reacts_to<atom("add"),int,int>::with<int>,
>>                     reacts_to<atom("hello")>::with<std::string>);
>> }
> I don't think I would use this kind of assertions.

Well, it could make it easier to debug an actor-based application. Another possibility is to perform static analysis (maybe using a Clang plugin).

>> Perhaps it would be possible to include the first presented approach for type-safe actors along with the (dynamic) default actor API. In this way, we could compose type-safe subsystems of actors whenever a problem can be solved using the limited API.
> Yeah, providing both approaches would be great.

Agreed. It should be straightforward to implement, too.

>>> While I find your tuple pattern matching quite interesting I'm looking
>>> for a more low level interface for the Actors part, that doesn't force
>>> to use tuples. It provides just a way to identify uniquely the message
>>> type and the associated data.
>>> 
>>> on<messageType> >> [](associatedDateType const& data) { /*...*/},
>>> 
>>> Of course the user could already use {atom("messageName"),
>>> associatedDataType()}, but this forces the user to use a cow data type
>>> that could be too costly for some applications. As most the
>>> communications are point to point, the ownership of the message is quite
>>> clear in these cases, so no need for COW.
>> I guess this is more of an optimization issue. We have to wrap the values into a mailbox_element anyways before putting the message to an actor's mailbox (http://libcppa.blogspot.de/2011/04/mailbox-part-1.html). This element might as well *be* the tuple in case of 1:1 communication. When forwarding the message, we then could simply transfer ownership of the mailbox element. Only when accessing self->last_dequeued(), we would have to make sure to provide COW semantics.
> When there are a lot of exchanged messages optimizing the communication 
> seems important to me.  I would prefer move semantics in this particular 
> case. COW doesn't provides the efficiency I'm locking for. This doesn't 
> mean that your approach is not coherent, but at a higher level.

libcppa already uses move whenever possible. As long as you send a message to one actor (using move on your arguments), you won't make copy of any value, unless you do something silly like 'send_tuple(buddy, self->last_dequeued())'.

>>> Last but not least your library works only on some C++11 compilers. I
>>> have nothing against having C++11 specific libraries in Boost, even the
>>> opposite (I proposed it long time ago) but the fact is that currently,
>>> the Boost libraries provide at least a minimal interface on C++98
>>> compilers. I don't know what the Boost community thinks about this point
>>> now.
>>> Welcome, keep up the good work and good luck with your library,
>>> Vicente
>> Thank you. At the "Future of Boost" session at the C++Now conference, most developers agreed that it makes sense to embrace C++11 and to use the momentum of the new standard. I hope this is a broad consensus among all Boost developers.
>> 
>> 
> Great. Glad to hear we are ready to move.
> 
> Best,
> Vicente