Re: [Boost-users] [Regex | Xpressive] Efficiently "grepping" large files

Subject: Re: [Boost-users] [Regex | Xpressive] Efficiently "grepping" large files
From: Chris Cleeland (chris.cleeland_at_[hidden])
Date: 2011-08-17 08:43:14

On Tue, Aug 16, 2011 at 3:44 PM, Thomas Luzat <thomas_at_[hidden]> wrote:
> The circular buffer needs to know when old data is no longer required,
> though, that is regex_search no longer works on it or backtracks into it.
> The best idea so far is to read blocks of data and do slightly overlapped
> calls of regex_search on those blocks....
> I would not mind to specify a maximum context of a few KB to maybe 1 MB, but
> I do not want to change semantics. Is there a better approach? I was
> thinking about supplying some intelligent iterators, but Regex/Xpressive
> might leave copies around longer than I anticipate, such that I still cannot
> decide where they are and how far they may still backtrack.
>
> If that's not really possible with the boost libraries I would be interested
> in alternatives, too. I am already considering looking into some grep
> implementations.

Have you considered mmap'ing the file and allowing all your activity
to occur on the mmap'd file?  That way the VM subsystem would worry
about paging things in or out as necessary, and there wouldn't be any
issues with contention across multiple threads. Of course, if you
don't have mmap on your system...