[boost] Re: Tokenizer Improvements

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2002-07-26 14:33:22

Hi, John

It took me some time to review tour changes. Everything seems acceptable.
But here I wanted to discuss more generic concern about tokenizer design.
IMO token iterator is in it's nature input iterator. Accordingly it should
behave like one. What I mean is that interface and usage should look like
this:

for_each( token_iterator(source ), token_iterator(), do_something );
or
for_each(  token_iterator( source, token_iterator_policy( ... ) ),
                token_iterator(),
                do_something );

I do not see any value in tokenizer class itself. To store shared
information we better use shared_ptr<token_iterator::Impl>. We could afford
another level of indirection in this case I think.

IMO it will make your design cleaner and more obvious

Gennadiy.

"UF" <jbandela_at_[hidden]> wrote in message
news:000601c22387$eb681820$26bfd43f_at_fgsqn...
> I have recently made some changes to tokenizer. Among the changes are
using
> tok.assign in some TokenizerFunctions and sharing 1 TokenizerFunction in a
> tokenizer with all its iterators. Please see
> http://plaza.ufl.edu/jbandela/tokenizer/ for more information as well as
to
> download and try the new changes. I would like some feedback on these
> changes before putting them into the CVS.
>
> Regards,
>
> John R. Bandela
>
>
> _______________________________________________
> Unsubscribe & other changes:
http://lists.boost.org/mailman/listinfo.cgi/boost
>