Re: [boost] [rpc] Introducing Boost.Reflect, Boost.CMT, and Boost.RPC

Subject: Re: [boost] [rpc] Introducing Boost.Reflect, Boost.CMT, and Boost.RPC
From: Daniel Larimer (dlarimer_at_[hidden])
Date: 2011-08-04 23:09:07

On Aug 4, 2011, at 9:23 PM, Julien Nitard wrote:
> 
> I have another use case :
> 
> After having written the byte swapper and csv output, the next logical
> application for reflection is a parser.
> The input is something of the form:
> {  { message_type = name }  |  key1=value1 | key2={ ... } |  key3=value3 |
> ...  }
> 
That looks very similar to json, put quotes around the keys and replace | with ','  and '=' with ':' and you have json.

Using my own JSON parser using Spirit.Qi (soon to be converted to utree from variant)
https://github.com/bytemaster/Boost.RPC/blob/master/include/boost/rpc/json/value_io.hpp

Using Json Spirit, using Spirit Classic 
https://github.com/bytemaster/Boost.RPC/blob/master/include/boost/rpc/json.hpp

I had an idea where I would 'auto-generate' a Spirit parser for an object that used the field names
as symbols and dumped the result right into the proper field.  This would then allow me to parse JSON objects
into c++ structs faster than anything else out there.  (no need for O(n) key lookup or temporary generic json container)

The downside would be the compile times associated with having so many spirit parsers.  

> 
> 
> All this to say that boost::reflect seems to have a lot of patterns ... and
> I think there's work to be done in the library to factor some boilerplate
> code, but I am not sure how it can be done (if at all). I think at least you
> could document the ones you think will be used more often. I also think that
> the calculator example though impressive, is a bit obscure for a newcomer to
> your library.

The serialization pattern is one that my RPC library will use in many places:
1) JSON
2) RAW (raw bytes, no special encoding), perhaps RAW with byte swapping 
3) Protocol Buffers 
4) XML  (several variations)
5) Conversion to Lua or Python objects in embedded scripting engines. 
6) Your CSV example

All of these have a very similar pattern with certain 'primitives' that require custom support and everything
else is based upon recursive calls to the reflector.  Finding a way to factor this pattern out would significantly
help development of my RPC library.

> 
> 
> The next thing that I'll be working on is that I need to map the names of
> the struct to their string counterpart as the first type of the parsing
> process, and here again it has to be done manually. That is I'll have to
> find a way to list my types in a map <string,type> while I am already
> calling BOOST_REFLECT that could do it at the same time (or at least I think
> it can, I didn't try yet). I think it is one pattern that would be a very
> useful addition.

I suspect that the following is not what you wanted: 

std::string name = boost::reflect::get_typename<type>() 

What I think you are asking for from the reflection library is some kind of factory that given a "type" will create a new type?

There are a lot of challenges with this (an older reflection library that I wrote was based upon such a system).  The first challenge is the location of the global variable, how does this interact with loading shared libraries.  Then there is the issue of when does this get populated.  You need to initialize a static variable for each type which means that the macro would need to be in the CPP file.... I believe I saw a trick for getting static initialization of variables in a header file via typedef of a template that contains the static initializer.   I have not been able to find that trick again.  One last detail is that any code that is not referenced is stripped by the static linker so even if you go through all of the trouble to create these static variables, they will be optimized out and never registered with the global factory unless you call a method in the object file that contains the static variable that registers your type with the factory.

If you add such a feature to the existing REFLECT() macro then it will result in users paying for something they do not necessarily want.  

Any solution to this problem will need to be a separate macro, BOOST_REFLECT_REGISTER(TYPE) to register the type with the factory.  Perhaps, it would also include information regarding available constructors...    Perhaps the REGISTER macro could include the more general REFLECT macro so you get down to only needing one macro.  This will only work if we can find a way to initialize static variables in the header.

Perhaps you were simply looking to register parsers and not constructors.  

> 
> Would you be interested if I worked on patch to do this ?
> 

It is a git repo... fork it and send me a pull request with anything you come up with.  If we can agree on a solution then I will be more than happy to integrate it with my library.   

Thanks again for your interest and feedback!