Proto for Boost mailing list page: [proto] looking for an advise

Subject: [proto] looking for an advise
From: Maxim Yanchenko (maximyanchenko_at_[hidden])
Date: 2010-12-27 05:02:42

Hi Eric and other gurus,

Sorry in advance for a long post.

I'm making a mini-language for message processing in our system.
It's currently implemented in terms of overloaded functions with
enable_if<match<Grammar>> dispatching, but now I see that
(a) I'm reimplementing Phoenix which is not on Proto yet in Boost 1.45.0 (that's
how I found this mailing list). It would be great to reuse what Phoenix has;
(b) I need to do several things on expressions and I don't know what would be
the best way to approach them all.

Here is a background.
Every message is a multiset of named fields (i.e. is a multimap
FieldName->FieldValue).
I have a distinct type for each FieldName, so I can do some multiprogramming on
sets of FieldNames, like making generating a structure that will hold values of
the fields I need, by list of field names (e.g. fusion::map).

While processing a message, I can do some checks like if particular field is
present, if it's equal or not to some value, if it matches a predicate etc.
They are implemented as a set of predicate functions "condition" like 

  template< class Msg, class Expr >
  typename boost::enable_if< proto::matches<Expr, proto::equal_to< proto::_,
proto::_ > >, bool >::type
  condition( const Msg& msg, const Expr& expr )

with various condition grammars in enable_if<matches<...>>

The processing itself is various operations that I can do to a message, e.g.
store it, send it, add more fields etc.
They are implemented as a set of functions "process_msg" like 

  template< class Msg, class Expr >
  typename boost::enable_if< proto::matches<Expr, SendToGrammar > >::type
  process_msg( const Msg& msg, const Expr& expr );

again, with various action grammars inside enable_if<matches<...>>.

All this naturally goes to syntax like if_(...)[...].else_[...] which I'd like
to reuse from Phoenix and throw my own implementation away.
If/else looks like:
  template< class Msg, class Expr >
  typename boost::enable_if< proto::matches<Expr, IfElseGrammar > >::type
  process_msg( const Msg& msg, const Expr& expr )
  {
    if ( condition( msg, IfElseGrammar::cond(expr) ) )
      process_msg( msg, IfElseGrammar::then(expr) );
    else
      process_msg( msg, IfElseGrammar::else_(expr) );
  }
where all getters are inside IfElseGrammar class (it's usual proto grammar).

Here is a sample of a functor:

  if_( any_field(Fields::Field1, Fields::Field2, Fields::Field3) == "test"
       && optional(Fields::Field4) == 3.1415
       && mandatory(Fields::Field5) == 1.41 )
  [
    append_field( Fields::Field6, 2.71 )
    && store( storage )
    && append_field( Fields::Field7, "qwe" )
    && send_to( client1 )
  ]
  .else_
  [
    send_to( client2 ),
    send_to( client3 )
  ]

Important notes:
1. As you can see, fields can be of different types (by default it's a string)
and we can request the same field for different value type at the same time
(e.g. both string and double)
2. mandatory(Fields::Field5) will throw if the field is not present, while
optional(Fields::Field4) will just return false for the equality check.
3. comma in the actions means, that both send_to will be applied to the same
message, while && means that the next action will receive a message from a
previous action, i.e. we will store the message with appended Field6, and will
send to client1 with both Field6 and 7.
4. In &&, for performance and exception handling reasons I can't return a
message from each elementary action, instead the next message will be called in
the context of the previous call. This actually needs a change of associativity
of operator&&.

Let me explain the point 4.
Processing function for a simple action (e,g, for sending) would have a
signature like

  template< class Msg, class Expr >
  typename boost::enable_if< proto::matches<Expr, SendToGrammar > >::type
  process_msg( const Msg& msg, const Expr& expr );

But if we want to append a field before sending, we need to execute a code like
this:
{
  MsgBuilderOnStack mb;
  mb.append(msg);
  mb.append(Fields::Field6, 2.71);
  send_to( mb.get_msg(), client1 );
}
As you can see, I need to call the next action (send_to) in context of the
previous one (append_field), and I do this by passing the next action to the
process_msg directly as another parameter Next:

  template< class Msg, class Expr, class Next >
  typename boost::enable_if< proto::matches<Expr, AppendField > >::type
  process_msg( const Msg& msg, const Expr& expr, const Next& next )
  {
    MsgBuilder mb;
    mb.append( msg );
    mb.append( value(child_c<1>(expr)), value(child_c<2>(expr)) );
    process_msg( mb.get_msg(), next );
  }

Then operator&& is written as:

  template< class Msg, class Expr, class Next >
  typename boost::enable_if< proto::matches<Expr, AndGrammar > >::type
  process_msg( const Msg& msg, const Expr& expr, const Next& next )
  {
    process_msg( msg,
      child_c<0>(expr),
      proto::make_expr< proto::tag::logical_and >( child_c<1>(expr), next) );
  }

Here I use make_expr<logical_and> to actually rewrite an expression to make it
right-associative: A && B && C becomes A&&(B&&(C)) by means of nested call:
"enter A, do something, enter B, do something, enter C, do something, exit C,
exit B, exit A".

So far everything works fine, except some things I'm not very comfortable with,
and I hope you will help:

(a) everything runs on enable_if. I expect it to become more concise and clean
if I use either transforms or contexts.

(b) a lot of Phoenix is basically reimplemented from scratch (thanks Eric, with
Proto it was very easy to do!). But I don't know how to extend Phoenix so it
could work in my expressions with my things like "any_field", "optional",
"mandatory" etc.

(c) I have two sets of functions: "condition" and "process_msg". Probably they
are not need given that almost everything from "condition" can be somehow reused
from Phoenix. OTOH, it's a clear separation of predicates and actions, so I can
use only "condition" part when I want to just check a message for some property,
without doing something to it. So maybe this separation is good, I'm not sure.
Probably I just need two separate grammars, and probably separate domains to
express each (because && has completely different meaning in condition (plain
logical and) and processing (queuing) context). And then I need to merge them
somehow to a full-blown check/process expression.

(d) (this is not implemented yet) I want to extract (in compile time, of course)
a list of fields and their types (can be many) and whether they are
optional/mandatory. Then I will use this information to build a structure that
will carry cached field values and pass this structure around together with the
message, to avoid expensive repetitive rescans of a message. This is probably
what transforms are for, because it looks like the calculator arity example. 

Thanks everyone who read until the end :)

Any input is greatly appreciated.

Thanks,
Maxim