Re: [boost] Update of TokenIterator

From: Aleksey Gurtovoy (alexy_at_[hidden])
Date: 2000-08-26 07:03:33

----- Original Message -----
From: <jbandela_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Friday, August 25, 2000 9:45 AM
Subject: [boost] Update of TokenIterator

> You can find an implementation that
> uses iterators in TokenIterator\TokenIteratorUpdate.zip

I really like your examples in TokenizerTest.cpp (BTW, 'iter' there actually
can be a const iterator). However, the implementation (in .hpp header) still
can be improved. My main concern is that classes in the header seem to be
quite complicated. Some of this complexity may be unavoidable because of the
nature of performed tasks (e.g., I agree that parsing code might look really
sophisticated), but probably some things can be improved in any case. For
instance, I think that splitting the header to at least 2 files - one with
definitions of some 'standard' tokenizers and another with the definition of
'token_iterator' itself, would probably help that task.
Actually, I would start not with code, but with definitions of two
concepts - TokenIterator and Tokenizer (or whatever :), because the
interface and requirements imposed (at least) by the second one are not
obvious for me now, and interaction between two of them is even more vague.
(well, actually I understand your code, but still...). After the concepts
will be defined, I think we would be able to clean up a lot of things in the
code.
I would take the current version of the TokenizerTest.cpp as a example of
simple use-cases of the class; probably it would be useful also to write a
couple of more sophisticated samples where token_iterator is supposed to be
useful (e.g. your calculator sample).

Sorry for not commenting the code - just don't think that's really needed at
this stage.

--Aleksey