Re: [Boost-users] [iostreams] regex_filter how-to

Subject: Re: [Boost-users] [iostreams] regex_filter how-to
From: Eric MALENFANT (Eric.Malenfant_at_[hidden])
Date: 2009-09-09 10:01:05

Micha³ Nowotka wrote:
>> IOStream's regex_filter loads the whole file in memory
> befory applying the regex on it.
> 
> No, it doesn't.

Yes it does. Or I misread the docs and code  :)
basic_regex_filter is an aggregate_filter. From IOStreams docs on aggregate_filter:

<quote from http://www.boost.org/doc/libs/1_40_0/libs/iostreams/doc/classes/aggregate.html>
The class template aggregate_filter is a DualUseFilter for use as a base class by Filters which filter an entire character sequence at once. Because a aggregate_filter must read an entire character sequence into memory before it begins to filter,[...]
</quote>

> As I said before, when you compile my code and use it against very
> large file in which all matching lines lay near the end then the
> output file will be empty. It can't load whole file because streams
> use buffers. 

That surprises me. Looking at the implementation of aggregate_filter (http://www.boost.org/doc/libs/1_40_0/boost/iostreams/filter/aggregate.hpp), I don't see any provisions for out of memory conditions, so I would have expected an attempt to read a several gigabytes file to result in a std::bad_alloc exception.