Re: [Boost-users] [iostreams] bzip2 filter fails with large file on Xen system

Subject: Re: [Boost-users] [iostreams] bzip2 filter fails with large file on Xen system
From: Nicola (nvitacolonna_at_[hidden])
Date: 2010-03-24 18:19:45

In article <4BAA33F4.4020406_at_[hidden]>, eg <egoots_at_[hidden]> wrote:

> On 3/24/2010 6:42 AM, Nicola wrote:
> > In article<4BA92A12.8030708_at_[hidden]>, eg<egoots_at_[hidden]>  wrote:
> >
> >> http://archives.free.net.ph/message/20070831.015329.594803cf.el.html
> >
> > Thanks for the link. All iostreams tests pass. But the tests use streams
> > of small size, as far as I can see, so I think that they don't cover my
> > issue.
> >
> 
> Can you post your test program? I can try to run it on a Windows box to 
> see if I can reproduce it.

As I've said, it's essentially the example from the Iostreams doc page:

#include <fstream>
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/bzip2.hpp>

int main(int argc, char * const argv[]) 
{
    using namespace std;
    using namespace boost::iostreams;

    ifstream file(argv[1], ios_base::in | ios_base::binary);
    filtering_streambuf<input> in;
    in.push(bzip2_decompressor());
    in.push(file);
    try {
      boost::iostreams::copy(in, cout);
    }
    catch (std::exception& e) {
       std::cout << "EXCEPTION CAUGHT: " << e.what() << std::endl;
    }
    return 0;
}

The output of the above program is

EXCEPTION CAUGHT: bzip2 error

when the argument is a “big” bzipped file (~270Mb).

The program I'm developing also fails to read the compressed file, but 
without errors: it just doesn't read anything. The following is a 
minimal self-contained excerpt that exhibits such behaviour:

io.hpp:

#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/bzip2.hpp>
#include <fstream>

class bz2istream :
public boost::iostreams::filtering_stream<boost::iostreams::input>
{
public:
   bz2istream(std::string const& path) : pBz2file(path.c_str(), 
std::ios_base::in | std::ios_base::binary)
   {
      if (pBz2file.fail())
      {
         // Here we are not distinguishing failbit from badbit...
         setstate(std::ios_base::failbit);
      }
      else
      {
         push(boost::iostreams::bzip2_decompressor());
         push(pBz2file);   

      }
   }
   
private:
   std::ifstream pBz2file;
};

main.cpp:

#include <cstdlib>
#include <iostream>
#include "io.hpp"

int main (int argc, char * const argv[]) {
        bz2istream bzippedfile(argv[1]);
        if (!bzippedfile) {
                std::cout << "Error reading file." << std::endl;
                return EXIT_FAILURE;
        }
        std::string field;
        while (bzippedfile.peek() != EOF) {
                bzippedfile >> field;
                std::cout << field << std::endl;
        }
        std::cout << "Finished.";
        endl(std::cout);
    return EXIT_SUCCESS;
}

The above also runs fine when the input file is a few kb long.

The input files are bzipped2 tab-delimited text files, whose content can 
be shown with bzcat on that system, so they are not corrupt. And I can 
process them just fine on another system with the code above.

Maybe there is some compiler flag or variable I should try to set on 
that particular system to compile the library and/or my code?

Nicola