Re: [Boost-users] Tokenizer Question

From: Robert Zeh (razeh_at_[hidden])
Date: 2005-06-13 08:37:12

"Tom Browder" <tbrowder_at_[hidden]> writes:

> The accumulated times (sec) are surprising:
>  
>                Boost    Victor's   Mine
>               =====    =====   ====
> no loop       1.50      38.84   20.13
> loop        131.91      38.89   23.91
>  
> Granted, I didn't do the tests multiple times, but it seems to me that the
> Boost tokenizer is great if you don't need to iterate through it, but it is
> the pits if you do.
>  
> -Tom
>  
> I'll send you my code and results if interested.
>  
> -Tom

  

I'd be interested in seeing profiling output from your test ---
when the tokenizer is reading from an input stream it builds up the
strings character by character with operator +=.

It can avoid creating the tokens character by character when it isn't
dealing with input streams (it checks for an input stream by checking
for std::input_iterator_tag, which I assume is set for istringstream).

Robert Zeh
http://home.earthlink.net/~rzeh