Boost logo

Boost Users :

Subject: Re: [Boost-users] Writing large binary files with boost gzip
From: Anders Knudby (knudby_at_[hidden])
Date: 2011-04-13 09:20:16

On Mon, Apr 11, 2011 at 3:35 AM, <Viatcheslav.Sysoltsev_at_[hidden]> wrote:

> Ok, here's my entire script. As mentioned in the comments it works fine if
>> size=5e5. It also runs fine if size=5e6, but then the resulting file is
>> corrupt (i.e. cannot be gunzipped with the gzip utility). Try it out. I'm
>> using Visual Studio 2008 in a Windows XP machine.
>> #include <boost/iostreams/device/file.hpp>
>> #include <boost/iostreams/filter/gzip.hpp>
>> #include <boost/iostreams/filtering_stream.hpp>
>> using namespace std;
>> namespace io = boost::iostreams;
>> int main()
>> {
>> //Set filename
>> string outfile = "c:/outfile.bin.gz";
>> //Set filesize
>> int size = int(5e6); // <- If I change this to '5e5' instead of '5e6',
>> everything works just fine.
>> //Declare memory block to be compressed to file
>> char* memblock = new char [size];
>> //Create a filtering_ostream out
>> io::filtering_ostream out;
>> //Assigns the gzip_compressor to out
>> out.push(io::gzip_compressor());
>> //Assigns out as a file sink
>> out.push(io::file_sink(outfile));
>> //Write memblock to out
>> out.write(memblock, size);
>> //Clean up
>> delete[] memblock;
>> io::close(out); //Note, also tried 'out.close();', 'io::close(out,
>> ios_base::out);' and 'close(out);'. Same result.
>> return 0;
>> }
> Works just fine on linux gcc 4.3.2 boost 1.45. with 5e5, 5e6 and 5e7 size.
> As a guess, I'd delete memblock after closing the out to be on a safe side,
> maybe it helps.
> -- Slava
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]

If anyone is still interested, this is what I got to work after much
hair-pulling. Maybe someone who understand streams (std and boost) better
than I can explain why this works and my previous attempts created corrupt
files when size was large (again, I'm using Windows, VS 2008). Cheers,

#include <fstream>
#include <iostream>
#include <sstream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>

using namespace std;
namespace io = boost::iostreams;

int main()
    //Trying with stringstream, which can act as both istream and ostream
    int size = 5700 * 4800 * 2; //This happens to be the actual size of the
data I need to write
    char* memblock = new char [size]; //Data to write

    stringstream ss(stringstream::in | stringstream::out |
stringstream::binary); //Declare ss
    ss.write(memblock, size); //Write data to ss

    io::filtering_streambuf<io::input> buf; //Declare buf
    buf.push(io::gzip_compressor()); //Assign compressor to buf
    buf.push(ss); //Push ss to buf
    ofstream out("c:/outfile.gz", ios_base::out | ios_base::binary);
//Declare out
    io::copy(buf, out); //Copy buf to out

    //Clean up
    delete[] memblock;

    return 0;

Anders Knudby, PhD
e-mail: knudby_at_[hidden]
skype ID: anders.knudby
Man's mind, once stretched by a new idea, never regains its original
~ Oliver Wendell Holmes

Boost-users list run by williamkempf at, kalb at, bjorn.karlsson at, gregod at, wekempf at