Re: [Boost-users] Writing large binary files with boost gzip

Subject: Re: [Boost-users] Writing large binary files with boost gzip
From: Anders Knudby (knudby_at_[hidden])
Date: 2011-04-13 09:20:16

On Mon, Apr 11, 2011 at 3:35 AM, <Viatcheslav.Sysoltsev_at_[hidden]> wrote:

> Ok, here's my entire script. As mentioned in the comments it works fine if
>> size=5e5. It also runs fine if size=5e6, but then the resulting file is
>> corrupt (i.e. cannot be gunzipped with the gzip utility). Try it out. I'm
>> using Visual Studio 2008 in a Windows XP machine.
>>
>> #include <boost/iostreams/device/file.hpp>
>> #include <boost/iostreams/filter/gzip.hpp>
>> #include <boost/iostreams/filtering_stream.hpp>
>>
>> using namespace std;
>> namespace io = boost::iostreams;
>>
>> int main()
>> {
>>    //Set filename
>>    string outfile = "c:/outfile.bin.gz";
>>
>>    //Set filesize
>>    int size = int(5e6); // <- If I change this to '5e5' instead of '5e6',
>> everything works just fine.
>>
>>    //Declare memory block to be compressed to file
>>    char* memblock = new char [size];
>>
>>    //Create a filtering_ostream out
>>    io::filtering_ostream out;
>>
>>     //Assigns the gzip_compressor to out
>>    out.push(io::gzip_compressor());
>>
>>     //Assigns out as a file sink
>>    out.push(io::file_sink(outfile));
>>
>>    //Write memblock to out
>>    out.write(memblock, size);
>>
>>    //Clean up
>>    delete[] memblock;
>>    io::close(out); //Note, also tried 'out.close();', 'io::close(out,
>> ios_base::out);' and 'close(out);'. Same result.
>>
>>    return 0;
>> }
>>
>>
> Works just fine on linux gcc 4.3.2 boost 1.45. with 5e5, 5e6 and 5e7 size.
> As a guess, I'd delete memblock after closing the out to be on a safe side,
> maybe it helps.
>
> -- Slava
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>

If anyone is still interested, this is what I got to work after much
hair-pulling. Maybe someone who understand streams (std and boost) better
than I can explain why this works and my previous attempts created corrupt
files when size was large (again, I'm using Windows, VS 2008). Cheers,
Anders

#include <fstream>
#include <iostream>
#include <sstream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>

using namespace std;
namespace io = boost::iostreams;

int main()
{
    //Trying with stringstream, which can act as both istream and ostream
    int size = 5700 * 4800 * 2; //This happens to be the actual size of the
data I need to write
    char* memblock = new char [size]; //Data to write

    stringstream ss(stringstream::in | stringstream::out |
stringstream::binary); //Declare ss
    ss.write(memblock, size); //Write data to ss

    io::filtering_streambuf<io::input> buf; //Declare buf
    buf.push(io::gzip_compressor()); //Assign compressor to buf
    buf.push(ss); //Push ss to buf
    ofstream out("c:/outfile.gz", ios_base::out | ios_base::binary);
//Declare out
    io::copy(buf, out); //Copy buf to out

    //Clean up
    out.close();
    delete[] memblock;

    return 0;
}

-- 
Anders Knudby, PhD
e-mail: knudby_at_[hidden]
skype ID: anders.knudby
Man's mind, once stretched by a new idea, never regains its original
dimensions.
~ Oliver Wendell Holmes