Re: [Boost-users] regex iterator question

Subject: Re: [Boost-users] regex iterator question
From: John Maddock (john_at_[hidden])
Date: 2009-03-30 11:47:51

> I've got a need for regex and I would like to use it to extract tokens 
> matching a regular expresssion from a file stream.  Seems like this would 
> be a common desire.
>
> So first shot is
>
> boost::regex_token_iterator<some_input_iterator>
>
> This doesn't work since regex_token_iterator requires a bidirectional 
> iterator.  Seems reasonable enough.
>
> So next I comb through boost and find
>
> #include <boost/spirit/iterator/multi_pass.hpp>
>
> and try
>
> boost::regex_token_iterator<
>    boost::spirit::multi_pass<some_input_iterator>
>>
>
> I'm thinking this is veeeeeery cool - maybe a 1000 lines of free code 
> included for the price of one.
>
> But, doesn't work.  multi_pass is a forward_trasversal iterator while 
> regex_token_iterator requires a bidirectional_trasversal_iterator.  A huge 
> disappoint to come soooo close.
>
> Thinking about it, this problem must come very often.  How is it usually 
> addressed?  There must be a simple bridge across this.  In a pinch, I'll 
> just have to load the whole file into some sort collection, but I prefer 
> the ultimate unlimited file size solution.

If you check the regex examples there are some "load_file" routines than 
dump a files contents into memory, but I agree it's not an ideal solution. 
I did experiment with some adapters to solve this issue in the early days of 
regex but never got a really good solution, and folks weren't demanding it 
so it got dropped :-(

But.. how about a memory mapped file?  Boost.Interprocess has support for 
that: 
http://www.boost.org/doc/libs/1_38_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.mapped_file 
although I admit it's not quite a one liner...

HTH, John.