Re: [Boost-users] count compressed bytes

Subject: Re: [Boost-users] count compressed bytes
From: Roland Bock (rbock_at_[hidden])
Date: 2010-06-13 10:01:15

Philipp Kraus wrote:
>
> On 2010-06-11 16:13:56 +0200, Philipp Kraus said:
>
>
> On 2010-06-11 15:45:42 +0200, Roland Bock said:
>
>
> Philipp Kraus wrote:
>
> On 2010-06-11 13:06:46 +0200, Philipp Kraus said: 
>
>
> On 2010-06-11 09:41:40 +0200, Roland Bock said: 
>
>
> Kraus Philipp wrote: 
>
> Hi, 
>
>
> I have implemented this litte code to compress a file 
>
>
> bio::filtering_streambuf< bio::input > l_deflate; 
>
> switch (m_compress) { 
>
> case gzip   : l_deflate.push( bio::gzip_compressor() );     break; 
>
> case bzip2  : l_deflate.push( bio::bzip2_compressor() );    break; 
>
> } 
>
>
>
>
> l_deflate.push( bio::file_source(p_str, BOOST_IOS::binary) ); 
>
>
>
> I need only the number of compressed bytes of l_deflate. I try to 
>
>
> use l_deflate.size() but it returns only a fixed number. 
>
>
>
> How I can read the number of compressed bytes? 
>
>
>
> Thanks 
>
>
>
> Phil 
>
>
>
> The only method I know is to add a small filter which counts the Bytes. 
>
>
> I have found this 
>
> http://www.boost.org/doc/libs/1_43_0/boost/iostreams/filter/counter.hpp 
>
> Can I use this class? I don't know how I can use the class (must I only 
>
> create a new object and push it to my l_deflate) ? 
>
>
> I have add this code: 
>
>
>        bio::filtering_streambuf< bio::input > l_deflate; 
>
>        switch (m_compress) { 
>
>            case gzip   : l_deflate.push( bio::gzip_compressor() );     
> break; 
>
>            case bzip2  : l_deflate.push( bio::bzip2_compressor() );    
> break; 
>
>        } 
>
>
>        // create counter structures 
>
>        bio::counter l_counter; 
>
>        l_deflate.push( boost::ref(l_counter) ); 
>
>        l_deflate.push( bio::file_source(p_str1, BOOST_IOS::binary) ); 
>
>
> but l_counter.characters() returns always zero. At next I have 
> declared a filter_ostream and copy l_deflate to the ostream in which I 
> pushed the l_counter. At this I run into a bus error. 
>
> For testing I've copy the l_deflate to std::cout and I can see the 
> compressed data. 
>
>
> Thanks for help 
>
>
> Hi,
>
>
> I haven't actually used this filter myself, but here are some thoughts:
>
>
>    1. when you get the counter to work, it will probably count the
>       uncompressed bytes. You should add it before the compressor, so
>       that bytes travel from source to compressor to counter.
>
>
> I swapped the lines
>
>
>    1. If l_counter.characters() is always zero, I'd suspect that the
>       filter is copied, when it is pushed to the stream. I haven't
>       found respective information in the documentation at once, so I
>       would suggest to add some debug code to the filter class, to 
>           * see if the characters are actually finding their way
>             through the filter
>           * make sure that the filter is not copied
>
> Once you figure out what went wrong, maybe you could create a 
> documentation ticket containing the relevant information.
>
>
> I tested both examples
>
> http://www.boost.org/doc/libs/1_43_0/libs/iostreams/doc/classes/counter.html#examples
>
> but in both the character-methods returns alway zero.
>
> In the second example there is also an error:
>
> io::filtering_ostreams out; is wrong it sould be io::filtering_ostream 
> out; without "s".
>
>
> At this time it is a very bad situtation. 
>
>
> I have found a solution:
>
>
>     io::filtering_istream in;
>
>     io::counter cnt;
>
>
>     in.push(boost::ref(cnt));
>
>     in.push(io::file_source("file.txt"));
>
>     
>
>     io::copy(in, std::cout);
>
>     io::close(in); 
>
>     
>
>     
>
>     std::cout << std::endl << std::endl << "Lines: " << cnt.lines() << 
> " Chars: " << cnt.characters() << std::endl;
>
>
> I must run the copy form the iostream, than I have the correct values 
> in characters() and lines(). But I don't want to use the std::cout. 
> I've found under 
> http://www.boost.org/doc/libs/1_43_0/libs/iostreams/doc/classes/null.html 
> a /dev/null device. At this time I don't know how to create a output 
> stream that declares a null-sink or null-stream
>
I attached a file I used to compare (de)compression performance. It uses 
output streams and sinks.

I also added two counters in the first function, which seem to work very 
well.

Regards,

Roland