Concepts for Token Iterator and Tokenizer

From: jbandela_at_[hidden]
Date: 2000-09-01 00:13:22

Thank you for your input. I have created some HTML files describing 
Token Iterator and Tokenizer concepts modeled after the SGI 
description of STL concepts. I am uploading them to the 
TokenIterator directory. Please look at them, and tell me what you 
think. Note that the posted code does not conform to that description 
yet (though it could easily be changed). I thought it might be better 
to nail down the concepts before actually changing the code to 
implement the described concepts.

Thanks

John Bandela

--- In boost_at_[hidden], "Aleksey Gurtovoy" <alexy_at_m...> wrote:
> ----- Original Message -----
> From: <jbandela_at_u...>
> To: <boost_at_[hidden]>
> Sent: Friday, August 25, 2000 9:45 AM
> Subject: [boost] Update of TokenIterator
> 
> > You can find an implementation that
> > uses iterators in TokenIterator\TokenIteratorUpdate.zip
> 
> I really like your examples in TokenizerTest.cpp (BTW, 'iter' there 
actually
> can be a const iterator). However, the implementation (in .hpp 
header) still
> can be improved. My main concern is that classes in the header seem 
to be
> quite complicated. Some of this complexity may be unavoidable 
because of the
> nature of performed tasks (e.g., I agree that parsing code might 
look really
> sophisticated), but probably some things can be improved in any 
case. For
> instance, I think that splitting the header to at least 2 files - 
one with
> definitions of some 'standard' tokenizers and another with the 
definition of
> 'token_iterator' itself, would probably help that task.
> Actually, I would start not with code, but with definitions of two
> concepts - TokenIterator and Tokenizer (or whatever :), because the
> interface and requirements imposed (at least) by the second one are 
not
> obvious for me now, and interaction between two of them is even 
more vague.
> (well, actually I understand your code, but still...). After the 
concepts
> will be defined, I think we would be able to clean up a lot of 
things in the
> code.
> I would take the current version of the TokenizerTest.cpp as a 
example of
> simple use-cases of the class; probably it would be useful also to 
write a
> couple of more sophisticated samples where token_iterator is 
supposed to be
> useful (e.g. your calculator sample).
> 
> Sorry for not commenting the code - just don't think that's really 
needed at
> this stage.
> 
> --Aleksey