Re: [boost] web service client based on boost?

From: Brian Davis (bitminer_at_[hidden])
Date: 2008-07-26 22:27:05

>I think it would be a great idea. But would you write this communications
>foundations layer ?

Take this one line:

Client->boost::iostream::HTTP->boost::iostream::SSL->ASIO->~network~->ASIO->boost::iostream:SSL->boost::iostream::HTTP->Server

Multiply by 5000

and that is about how many lines of code if not more this would take

This is also entering the realm of frameworks and arcitectures and
inevitably "cat herding" and bicycle shed discussions.  Try opening a socket
to a HTTP server in JAVA (yea I know I don't want to go there either) and
then try the same in C++... standard lib... standard architecture? It
becomes in C++ may hours of searching source forge and trying out
libraries.. reading various documentation and learning many different
software architectures then picking the right one then of course there are
the bugs... hopefully you chose one opensource or you could be in for a
world of hurt.

>From Boost.IOStreams lib Filters section
"The third Filter helper,
symmetric_filter<http://www.boost.org/doc/libs/1_35_0/libs/iostreams/doc/classes/symmetric_filter.html>,
is useful for defining filter based on C-language API such as zlib, libbz2
or OpenSSL"

which (OpenSSL)  is what  spawned off the idea for web services using
streams and filters.

Can't say I would be the best person for the job, but sure I am interested.
I was looking at iostreams and asio  for use as a dataflow model which
Stjepan has already created.  My interest in asio was/is two fold 1)
extending ASIO for RS232 and other asynchronous IO 2) using iostreams
(possibly) for database to preform ODBC connections to remote databases.  I
am not talking about writing an entire database (with the exception of as
simple text/file based database provided as a default), but rather providing
a front end / back end design one that allows any database to be connected
to on the backend and a standard C++ front end one that takes advantage of
all the boost goodies.  Initially I looked iostreams for writing a simple
csv filter for file operations to create a table of data then I looked at it
for full blown data base interface.  This is were iostreams start to break
down when they are used for more than simple streaming of characters and
other concepts are needed.  There is a common theme between WebServices,
Database, and XML  The data in the stream is non uniform (changing) or not
of a specific format.  This data must be coerced into type safe containers
that can be iterated through.  Transactions can be Asynchronous even for XML
where DTD and linked xml files may be on a remote server and a request may
not complete instantaneously and blocking is not the answer..

There is a common themes between Database and WebServices
- Transactions
- Manage non uniform data or changing data formats
- Asynchronous event notification
- The need to convert incoming data to type safe containers (fusion may be
the key here)
- The need to be be able to iterate and search data.
- The need to configure how the data flows through the program (i.e. is the
stream encrypted using OpenSSL as my one liner at the top shows).

Consider the PSEUDO code (I stress by no means have I started this.... it is
shear contemplation.   Database is also another topic I brought up at
BoostCon 2008) :

// Where database is a boost:iostream::BidirectionalDevice
boost::database myDB;
// Where query is a boost::iostream::filter::BidirectionalFilter
boost::database myQuery( "SELECT [Employee ID], [Employee Name] FROM
[Employees]" );

std::string employee;
boost::uint32_t employee_id;

// Since Database is a stream data base can be modified using SQL commands,
but only thoes which affect a database structure such as CREATE TABLE, DROP
TABLE etc"
myDB << "ODBC;127.0.0.1:1234,user=myuser, password=mypw" << std::endl;

// or
myDB.open( "ODBC;127.0.0.1:1234,user=myuser, pw=mypw"  );

// or for default boost text or binary based boost::database implementation
boost::filesystem::path = "/opt/var/mydb";

// Open and create if not in existence.
myDB.open(  path, ios::binary | std::ios::trunc  );

// hey maybe we could also change the file storage format filter.
myDB.push( csvFilter );

// Add the query filter to the database.
mydb.push( myQuery );

while( !myDB )
{
   myDB >> employee_id >> employee;

}

I am not certian how using streams for databases would fully look.  Maybe a
stream for each query or table object.  Maybe the database object would not
be a stream object.   In  my example down stream filters could change the
underlying device(source/sink) which would change the data retrieved at the
source (database) and would not actually "filter" the full data coming in
the stream.  I think this deviates from the IOStream design where each down
stream filter receives all data and filters (transforms) it accordingly.
Though sending all the data to each filter is possible and still an option
here, though it is not as efficient as it could be.   So the full table
would not be streamed to the query, but rather the query filter would change
the database device to select just the two columns of data in the database
table Employees and stream the result.  There's nothing saying that the
filter stream must modify the data as it passes through... it could modify
the base stream object itself either way the result should be the same.  Of
course this probably entering the realm of "shoe horning" iostreams to meet
database needs.

Consider  mydb.push( myQuery) call.  The push can be overloaded to get from
the query filter string.

The cautious would remember the following from iotreams lib doc:
  "Non-blocking Devices do not interact well with standard streams and
stream buffers, however, so most devices should be
Blocking<http://www.boost.org/doc/libs/1_35_0/libs/iostreams/doc/concepts/blocking.html>.
*See* Asynchronous and Non-Blocking
I/O<http://www.boost.org/doc/libs/1_35_0/libs/iostreams/doc/guide/asynchronous.html>
."

And we all know databases especially remote ones do not return transactions
immediately.  Which may require stream callbacks notifying when a
Asynchronous I/O data transaction request has fully completed and data is
ready.

Most of boost, to me personally (and I likes me the boost :-) ), seems like
a collection of basic low level APIs which the programmer can use to
construct what ever software they need.  Lets call these Tier1. Tier2  could
be the extension and use of these Tier1 libraries to create architectures to
really "boost" application developers to assist in creating web services,
database applications, GUIs (cppgui) etc.

I also asked at BoostCon for the 10 step process for submitting a library
and if you go to:

http://www.boost.org/development/submissions.html <--- Boost should really
make this a big green button next to the "get boost" button so say  "develop
boost" or in the time honored tradition "set boost"

I read:
"A bit of further description or snippet of code may be helpful. Messages
should be plain text; not rich text, HTML, etc"

so I submit these and other concepts.  If there is enough interest then sure
I can help or possibly lead the way. I am still just getting started.  My
current focus is on dataflow and databinding.

There is currently a submission on databases in the vault which was posted
by Nicola Musatti which provides much better type checking that streams
would provide, but provides no support for streams.

http://www.boostpro.com/vault/index.php?PHPSESSID=dc8a8cca1f95373201ace3827a3827b9&direction=0&order=&directory=Database

Nothing on the radar for web services that I can see in vault, trunk, or
sandbox.  So maybe it's time to look into Boost.WebServices.  I would like
to know what the road map is for Architectures in boost (database, gui,
webservices, XML, etc).

Looks like Dean Berris has a great start:

https://cpp-netlib.svn.sourceforge.net/svnroot/cpp-netlib

>From client.cpp in cpp-net-lib

>>
            // use Asio's iostream interface to perform
            // a synchronous request for data via HTTP
            // and then craft a response object to be returned
            // based on the headers received, and the contents
            using namespace boost::asio;

            ip::tcp::iostream socket_stream(request_.host(),
boost::lexical_cast<typename tag::str_type>(request_.port()),
ip::tcp::resolver_query::numeric_service);
            socket_stream
                << "GET /"
                << request_.path()
                ;
<<

shows that net-lib is directly bound to ASIO making SSL encryption difficult
as depicted by:

Client->boost::iostream::HTTP->boost::iostream::SSL->ASIO->~network

I am not saying that net-lib is bad, matter of fact I found it quite slick,
just that it does not support the kind of stream chaining I was talking
about.  Maybe this is not desired.

Brian