Re: [boost] web service client based on boost?

From: Dean Michael Berris (mikhailberis_at_[hidden])
Date: 2008-07-29 04:23:49

Hi Brian,

On Sun, Jul 27, 2008 at 10:27 AM, Brian Davis <bitminer_at_[hidden]> wrote:
>>I think it would be a great idea. But would you write this communications
>>foundations layer ?
>
> Take this one line:
>
> Client->boost::iostream::HTTP->boost::iostream::SSL->ASIO->~network~->ASIO->boost::iostream:SSL->boost::iostream::HTTP->Server
>
> Multiply by 5000
>
> and that is about how many lines of code if not more this would take
>

This assessment may even be conservative. Considering that you're
going to be making these generic pieces try to work together in a
matter that doesn't break, then you've got yourself a very large scale
project that although a lot of people want, not a lot of people have
the time/resources/motivation to write.

> This is also entering the realm of frameworks and arcitectures and
> inevitably "cat herding" and bicycle shed discussions.  Try opening a socket
> to a HTTP server in JAVA (yea I know I don't want to go there either) and
> then try the same in C++... standard lib... standard architecture? It
> becomes in C++ may hours of searching source forge and trying out
> libraries.. reading various documentation and learning many different
> software architectures then picking the right one then of course there are
> the bugs... hopefully you chose one opensource or you could be in for a
> world of hurt.
>

I know how painful this is. I think the first step (which is under way
IIRC) is to develop a common base for everything network-related in
C++ -- this I think is what Asio gives, and is already being
considered to be made part of the next version of the C++ standard.
That being said, what we have now are the building blocks to be able
to come up with more specific targetted tools.

Having said this, I don't necessarily think this should be made part
of Boost. Anybody should be able to come up with their own Web
Services framework in C++ and license it under the Boost Software
License without having to be part of the Boost C++ Library.

> >From Boost.IOStreams lib Filters section
> "The third Filter helper,
> symmetric_filter<http://www.boost.org/doc/libs/1_35_0/libs/iostreams/doc/classes/symmetric_filter.html>,
> is useful for defining filter based on C-language API such as zlib, libbz2
> or OpenSSL"
>
> which (OpenSSL)  is what  spawned off the idea for web services using
> streams and filters.
>
> Can't say I would be the best person for the job, but sure I am interested.
> I was looking at iostreams and asio  for use as a dataflow model which
> Stjepan has already created.  My interest in asio was/is two fold 1)
> extending ASIO for RS232 and other asynchronous IO 2) using iostreams
> (possibly) for database to preform ODBC connections to remote databases.  I
> am not talking about writing an entire database (with the exception of as
> simple text/file based database provided as a default), but rather providing
> a front end / back end design one that allows any database to be connected
> to on the backend and a standard C++ front end one that takes advantage of
> all the boost goodies.  Initially I looked iostreams for writing a simple
> csv filter for file operations to create a table of data then I looked at it
> for full blown data base interface.  This is were iostreams start to break
> down when they are used for more than simple streaming of characters and
> other concepts are needed.  There is a common theme between WebServices,
> Database, and XML  The data in the stream is non uniform (changing) or not
> of a specific format.  This data must be coerced into type safe containers
> that can be iterated through.  Transactions can be Asynchronous even for XML
> where DTD and linked xml files may be on a remote server and a request may
> not complete instantaneously and blocking is not the answer..
>

WRT to Databases, have you seen SOCI?

XML Stream parsing is something that's fit for the Iostreams lib along
with Asio underneath -- very important for XMPP and other streamed XML
protocols -- but I have very little experience with XML Stream parsing
and Iostreams to be able to give a better idea about it. But it does
sound natural to map stream-based protocols to the Iostreams
interface.

[snip]
>
> Most of boost, to me personally (and I likes me the boost :-) ), seems like
> a collection of basic low level APIs which the programmer can use to
> construct what ever software they need.  Lets call these Tier1. Tier2  could
> be the extension and use of these Tier1 libraries to create architectures to
> really "boost" application developers to assist in creating web services,
> database applications, GUIs (cppgui) etc.
>

Right. I think also the goal of Boost is to be a place for developing
software/libraries not just for general use, but also for eventual
standardization. Already, libraries like Regex, Thread, Tuple, and
(IIRC) Asio are going to be part of the next version of the standard.

[snip]
>
> Nothing on the radar for web services that I can see in vault, trunk, or
> sandbox.  So maybe it's time to look into Boost.WebServices.  I would like
> to know what the road map is for Architectures in boost (database, gui,
> webservices, XML, etc).
>
> Looks like Dean Berris has a great start:
>
> https://cpp-netlib.svn.sourceforge.net/svnroot/cpp-netlib
>
> >From client.cpp in cpp-net-lib
>
>>>
>            // use Asio's iostream interface to perform
>            // a synchronous request for data via HTTP
>            // and then craft a response object to be returned
>            // based on the headers received, and the contents
>            using namespace boost::asio;
>
>            ip::tcp::iostream socket_stream(request_.host(),
> boost::lexical_cast<typename tag::str_type>(request_.port()),
> ip::tcp::resolver_query::numeric_service);
>            socket_stream
>                << "GET /"
>                << request_.path()
>                ;
> <<
>
> shows that net-lib is directly bound to ASIO making SSL encryption difficult
> as depicted by:
>
> Client->boost::iostream::HTTP->boost::iostream::SSL->ASIO->~network
>
> I am not saying that net-lib is bad, matter of fact I found it quite slick,
> just that it does not support the kind of stream chaining I was talking
> about.  Maybe this is not desired.
>

Actually, the interface and implementation will change -- and is
currently being developed (by me) offline, and at a snail-steady pace.
The idea is that whether you use SSL or plain TCP, it will use Asio
underneath. There will be two client implementations: a synchronous
one, templated to whether you use SSL or use a proxy, etc.; and an
asynchronous one, supporting pipelining and "automatic session
management" with cookie and (eventual) caching support -- all while
trying to be a header-only implementation.

The message framework that's already there allows for building opaque
messages, and a transformation layer is provided to support
conversions from one message type to another either through library
implemented routines or through user-generated functions. That allows
you to:

1. Create a soap::message implementation that converts to an
http::message or an xmpp::message (whichever you support)
2. Have that object instance passed to a client that talks the
appropriate protocol
3. Get a generic protocol::message response
4. Convert to the appropriate (in this case soap::message) type for
further processing.

This allows me to write an HTTP(S) client without having to think
about all the possible messages that come through and only support one
kind and work well with it.

The message framework is already done, and help with testing and
further development would be greatly appreciated. :)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.