[boost] Interest from BoostCon

From: Preston A. Elder (prez_at_[hidden])
Date: 2007-05-30 15:04:04

Hey,

There was some interest in some code I had written a long time ago
expressed at BoostCon - I believe I linked this before, but the location
has changed.  Specifically these are libraries inside of Mantra.

These have not been modified in 2 years, so I wanted to know if there was
any interest in them (and as such cause for me to update and submit them).
If nobody is interested in them I will probably drop them, if people are
interested I can look into cleaning them up and submitting.

- boost::unformat - a logical reversal of boost::format (ie. scanf for
boost::format):
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/core/unformat.h

Right now this only unformats into strings, and is not extensively tested,
but it DOES work.  If there is interest I will enhance it to unformat int
a tuple, or something else (suggestions)?

- Intrusive Tracing - the ability to activate various levels/styles of
code-based tracing at runtime, and by logical code 'section'.
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/core/trace.h
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/core/trace.cpp

It uses macros - and I would like to expand it to more and better tracing
capabilities, but what is there works and is VERY convenient.  It even
supports a 'transactional' mode of tracing, to avoid trace cross-talk. 
Tracing can be completely compiled away for production code.

- Extension to program_options to allow for more robust checking of
provided values.
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/core/typed_value.h

Some of this functionality has been put into program_options, but not as
nice as I'd like it (at least last I looked at it).  This allows you to
do things like validate the value provided (must match this regex, must
fall within this range, must pass this check function).  It also allows
for more advanced things like value mapping (user uses value of 'INFO',
which gets mapped to the enum LogLevel_t::INFO or whatever).  It also
allows 'chained' checks.  This is probably showing its age though,
there have been certain enhancements to program_options that makes some of
the stuff here unnecessary.

- Date/Time and Duration formatting (both to and from) functions:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/core/datetime.h

These were written before the date/time facets - but they are still useful.

- Guarenteed event mechanism:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/service/events.h
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/service/events.cpp

This handles its own thread pool (ie. minimum number of standby threads),
and operates in such a way that it will guarentee an event is started
EXACTLY when you want it, even if it has to spawn a new thread to do so.

- In-process watchdog:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/service/watchdog.h
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/service/watchdog.cpp

Relatively simple, but saves everyone else writing one :)

- Various dispatching methodologies:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/service/messagequeue.h
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/service/its_messagequeue.h
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/service/routed_messagequeue.h
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/service/ots_messagequeue.h

These are message queues for different purposes.  Each (except OTS) is
backed by a thread pool that can resize based on high/low water marks.
* MessageQueue - Single queue for a pool of threads.
* ITS_MessageQueue - Implements a 'queue per thread' (with a 'transport'
thread) to reduce lock contention by delivering messages to a worker
thread for processing in bulk (includes 'fair' queueing).
* Routed_MessageQueue - Adds a key to the message, guarenteeing that only
one thread can process messages for a single key at any one time.
- OTS_MessageQueue - Reverses ITS_MessageQueue, made for the 'other' side,
ie. multiple threads sending data to a single thread (eg. a store and
forward thread or something).  In this case it only has a single thread
(plus a collector), and has a queue per thread that PUSH's onto it instead.

- Networking Stuff:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/net
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/net

Written before ASIO was accepted - probably not required anymore, but it
IS a simpler approach (ie. 3 classes, thats it).  Supports
select, poll and epoll.

- Generic Data Storage Backend:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/storage
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/storage

This allows you to have a common interface to your storage back end,
regardless of whether its an SQL database, Berkeley DB, XML file, etc.  It
also allows you to CODE a query, eg:
        Condition<C_LessThan>("col1", 3") &&
                (Condition<C_EqualTo>("col2", "hello") ||
                 Condition<C_EqualTo>("col2", "world")) &&
        Condition<C_Masked>("col3", FLAG_A | FLAG_B)

This will then expand to an SQL expression for SQL DB's, or apply to data
rows for non-SQL DB's - ie. it will work for all data sources.

Right now I plan on re-writing this anyway to support:
* Cross-table/database queries, better column limits and more operations:
(http://www.neuromancy.net/mantra/trac/ticket/1)
* Transactions & Cursors (http://www.neuromancy.net/mantra/trac/ticket/2)
* An Object-based model (http://www.neuromancy.net/mantra/trac/ticket/3)

But I would like to know if there is interest for this in the boost
community or more specifically, in turning it into a boost module in its
own right.

- Store and Forward and Disk-Overflow:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/file/saf.h
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/file/saf.cpp
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/file/filebuffer.h
and
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/src/file/filebuffer.cpp

The SAF mechanism is there to implement guarenteed delivery of data, even
if your application crashes.  The FileBuffer provides disk-based overflow
for memory-constrained situations (eg. where you have a socket, and you
can get massive bursts of data to go out the socket, but cannot send it as
fast as the burst - especially where you have thousands of such bursty
connections).

- Python bindings for some boost libraries:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/python/boost

Right now I have bindings for boost::format, date_time and tribool.  There
was interest in these but nowhere to put them.  In addition they need
updating (especially date_time).

I also plan on re-writing my Logging library:
http://www.neuromancy.net/mantra/trac/browser/trunk/Mantra-I/include/mantra/log
to be MUCH more efficient and flexible with regards to adding new logging
back ends and formatting (http://www.neuromancy.net/mantra/trac/ticket/4).

Sorry for the length of this post.  Please let me know if you think some
of these things would look good with a boost:: prefix :)

PreZ :)

BTW - I enjoyed BoostCon immensely, thanks again to the organizers and I
enjoyed meeting many of you.  This is an email you can contact me at :)