Re: [Boost-users] concat files for compression

Subject: Re: [Boost-users] concat files for compression
From: Roland Bock (rbock_at_[hidden])
Date: 2010-06-15 12:08:50

Kraus Philipp wrote:
> 
> Am 13.06.2010 um 15:41 schrieb Roland Bock:
> 
>> Philipp Kraus wrote:
>>> On 2010-06-11 09:34:27 +0200, Roland Bock said:
>>>
>>>> Kraus Philipp wrote:
>>>>> Hi,
>>>>>
>>>>> on my previous posting I have another questing:
>>>>>
>>>>> I create my compression stream with
>>>>>
>>>>> bio::filtering_streambuf< bio::input > l_deflate;
>>>>> switch (m_compress) {
>>>>> case gzip   : l_deflate.push( bio::gzip_compressor() );
>>>>> break;
>>>>> case bzip2  : l_deflate.push( bio::bzip2_compressor() );
>>>>> break;
>>>>> }
>>>>>
>>>>> For a single file I push it to zip with:
>>>>>
>>>>> l_deflate.push( bio::file_source(p_str1, BOOST_IOS::binary) );
>>>>>
>>>>> I need a concatination for two files like:
>>>>> l_deflate.push( bio::file_source(p_str1, BOOST_IOS::binary) );
>>>>> l_deflate.push( bio::file_source(p_str2, BOOST_IOS::binary) );
>>>>>
>>>>> for unix shell I do this with:
>>>>>
>>>>> cat file1.txt > tozip.txt
>>>>> cat file2.txt >> tozip.txt
>>>>>
>>>>> How I can push a concatinated file to the deflate stream buffer?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Phil
>>>>
>>>> Hi,
>>>>
>>>> I see two options:
>>>>
>>>> a) You write a new multi_file_source class which can any number of input
>>>> files.
>>>>
>>>> b) Instead of working with a source, you could work with a sink at the
>>>> other end of the pipeline. Then just copy the content of the two files
>>>> into the stream.
>>>
>>> Hi,
>>>
>>> I think I would like to use the second idea, but how I can do this?
>>>
>>> Thx
>>>
>>> Phil
>> Hi,
>>
>> currently, all your examples are using filtering input streams with a
>> source. For concatination and compression I would use a filtering
>> output stream and a sink (for instance a file sink).
>>
>> Then you open an ifstream for the first file and copy its content into
>> the filtering output stream.
>> Then you open an ifstream for the second file and copy its content
>> into the filtering output stream.
> 
> Thanks that was a very big hint.
> 
> My code is now:
> 
> bio::basic_null_sink<char> l_out;
> bio::counter l_counter;
> bio::filtering_streambuf< bio::output > l_deflate;       
> 
> l_deflate.push( bio::gzip_compressor() );
> l_deflate.push( boost::ref(l_counter) );
> l_deflate.push( l_out );
> 
>                   
> 
> std::ifstream l_file( p_str1.c_str(), std::ifstream::in |
> std::ifstream::binary );
> bio::copy( l_file, l_deflate );
> l_file.close();
> 
>             
> 
> l_file.open( p_str2.c_str(), std::ifstream::in | std::ifstream::binary );
>  ===> bio::copy( l_file, l_deflate ); <==== error
> l_file.close();
> 
> It works perfectly, except the second copy. On this line I get a
> "segmentation fault" (sorry I can't use the debugger, because the code
> runs on another system).

Glad to read that you made some progress :-)

I am not very good at reading the boost::iostreams code, but I think
boost::iostreams::copy closes the streams after completion. Thus, when
you try to copy the content of the second file into deflate stream, it
crashes (an exception would have been friendlier...).

You could try to use std::copy like this (untested code):

copy(istreambuf_iterator<char>(l_file), istreambuf_iterator<char>(),
ostreambuf_iterator<char>(l_deflate));

You need to close l_deflate by hand, then (your counter won't be correct
before you do, because gzip has to perform some buffering).

Regards,

Roland