Boost logo

Boost :

From: jbandela_at_[hidden]
Date: 2000-09-13 12:04:14

> It is referenced in the main tokenizer.hpp file since (a version of)
> punct_space_tokenizer is used as the default type in the second
> argument of the token_iterator class. I think token_iterator
should be just
> another sample, and not the default tokenizer. It's highly likely
that some
> other tokenizer will be used, so not including
punct_space_tokenizer and its
> owning header file means clients who don't use it won't pay for
having it.

I agree with you there. What you do not use, you do not pay for is an
underlying principle of C++. However, I believe we can leave the
default TokenizerFunc = punct_space_tokenizer<Iter> argument in
token_iterator, and remove the #include statement. If they want to
use the default, they can just include the punct_space_tokenizer
file. Otherwise, they can just leave it out and specify another

> Maybe some sort of traits class should be added. The
> would use that traits class, and so can any other potential
tokenizer or
> adapter.

I am not sure, I exactly understand what you saying. Would you please
provide some sample code that illustrates this.

> It choked on me, see below.

Go ahead and add a default, do-nothing constructor to the tokenizer.
It probably won't hurt anything. One thing about this tokenizer, is
that it is just a sample, and does not correctly handle the boundary
conditions where either what,with or both are empty strings. As this
is just a demo and not production code, I have not felt it was too
bad. However, I have created an updated version that fixes this
problem. I may post it later, if enough people ask me to.

Thanks for your input, and let me know if you can compile it with

John R. Bandela

Boost list run by bdawes at, gregod at, cpdaniel at, john at