[Boost-users] boost::TokenizerFunction

From: Zenaan Harkness (zen_at_[hidden])
Date: 2003-09-21 11:26:27

TokenizerFunction "Expression Semantics" table (documentation - eg. on
debian: /usr/share/doc/libboost-doc/HTML/libs/tokenizer/index.htm),
states that "tok is constructed", as part of "Precondition".

However I note two things:

1: on looking at the source code for the provided example functions,
each constructs a Token as it's first (or second) line of code in the
call to operator().

2: the "Semantics" says "The return value [bool] indicates whether a new
token was found in the sequence".

If there is no token available (eg. we are already at end of
stream(/string) and will return false) why waste effort constructing
storage for a token?

Why does the doco say that the token must be constructed as a
precondition? Is the code wrong, or the doco - ie. should my custom
function construct a new (empty) token, or follow doco and have user
construct one? It would seem potentially more efficient for the user to
construct one, as they may be reused.

I am trying to use (my own custom) function directly, in a for loop. Why
would I use tokenizer class to wrap it? There is no example on using a
function directly, thus these string of questions which I would like
some answers on.

The documentation for TokenizerFunction doesn't seem to have a proper
doxygen-like api description - at least not on my local debian
installation. So I have to revert to the source. As I'm a newbie, it is
requiring quite a bit of going back and forth to try and figure it all
out - eg., should I use a tokenizer (container), token_iterator, or just
for-loop on a specific (custom) tokenizer function??

Finally, as I've been trying to figure out how to write my own
TokenizerFunction. Delving into the code for the example TFs has
eventually led me there, but a simpler example in the documentation
would be very useful for other newbies, I'm sure. - I know, I should
post the code. Well, I'm not done yet, and I'm implementing a function I
need specifically, as opposed to just for learning or doco, but perhaps
later in the week I can send something along if no one beats me to it.
Once I understand it all a bit better.

Simple one-hit functions can be a good way to learn how to code what you
need (ie. not templatized). A progression from a one-hit to a
templatized version would be ideal for learning.

TIA
Zen

-- 
Mr Zenaan Harkness
Phone [phone]
Please respect the confidentiality of this email.