Re: [boost] Interest in simple unformatted binary stream I/O?

Subject: Re: [boost] Interest in simple unformatted binary stream I/O?
From: Marsh Ray (marsh_at_[hidden])
Date: 2011-04-29 01:14:55

On 04/28/2011 10:15 PM, Scott McMurray wrote:
> On Sun, Apr 24, 2011 at 18:43, Marsh Ray<marsh_at_[hidden]>  wrote:
>>
>> Rather than a machine and compiler-dependent binary IO facility based on
>> iostreams, I'd rather have a simple, minimal, portable binary streaming
>> interface that handled endianness and alignment well for wire-representation
>> defined structures.
>>
>> <iostream>  brings in a huge header tree with 99% unnecessary functionality
>> for binary protocols. The more efficient the headers, the better. Functions
>> that didn't need to read could just include the header for writing, and
>> vice-versa.
>>
>
> Would you be fine with it working on streambufs?  That way you'd only
> need<streambuf>...

The last time I messed with streambufs (some years ago) I came to the 
conclusion that they could not be accessed multithreadedly in a natural 
way. I tried to define my own custom ostream channel and it took 
hundreds of lines of code to get something that just functional and not 
optimized. IIRC, streamsize was defined to be a signed int and not 
64-bit clean.

But after preprocessing, <streambuf> adds only 1320 non-comment 
non-blank lines in addition to the 6529 I get just from including 
<string> which I don't try to live without.

Going all the way with <fstream> and <iomanip> it come to 13783 NCNB 
lines, although they take just 200 ms to parse on g++ 4.6.0.

Although I complain about header size and compile time, my problem is 
probably just conceptual. Probably I'm subconsciously terrified of 
locale dependencies or something.

All I really need is:

   // Binary write operation. May block.
   // Throws something derived from std::exception if output is
   // known to have failed.
   typedef boost::function<
             void(uint8_t const * begin, uint8_t const *end)
       > write_data_fn_t;

So my code can write:

   void some_code_emitting_bytes(write_data_fn_t data_out)
   {
       vector<uint8_t> buf(256, 0);

       // ... code involving buf ...

       data_out(&buf[0], &buf[0] + buf.size());
   }

Actually, the other day I wrote something slightly more complicated. It 
went like this:

// A function object representing write operations on an output stream.
//
// Intended to be an easy to use interface code needing to write data.
// Several overloads are provided for easy writing of common source data 
types.
//
// It should be cheap to copy, so functions may accept it as a non-ref 
parameter.
//
// This type is also intended to be easy to construct on-the-fly at the 
call site,
// even implicitly. The caller need only supply a function-like object 
taking a
// begin and one-past-end char pointer pair and throwing something 
derived from
// std::exception on error.
// The implementation should avoid gratuitous copy operations, but 
callers may
// still wish to attempt to write data in reasonably-sized chunks to avoid
// excessive function call overhead.
struct write_data_fxr
{
    //----- Construction.

    // Construct this type from an even simpler function object interface.
    // Intended to provide an implicit conversion from compatible functions.
    typedef boost::function<void (char const * beg, char const * end)> 
write_data_fn_t;
    write_data_fxr(write_data_fn_t fn);

    //----- Writing data.

    // Even though these operations don't modify the object per se, they 
are declared
    // non-const on the theory that they are non-idempotent with respect 
to the
    // underlying output stream.
    // These operations throw something derived from std::exception to 
indicate error.
    // Whether or not the operation is re-tryable or not is unspecified.
    //
    void operator () (std::string const &);
    void operator () (std::vector<signed char> const &);
    void operator () (std::vector<unsigned char> const &);
    void operator () (  signed char const * beg,   signed char const * end);
    void operator () (unsigned char const * beg, unsigned char const * end);

private:

    write_data_fn_t fn_;
};

Allowing my code to look like:

   void some_code_emitting_bytes(write_data_fxr data_out)
   {
       vector<uint8_t> buf(256, 0);

       // ... code involving buf ...

       data_out(buf);
   }

- Marsh