Boost logo

Boost :

From: David Abrahams (dave_at_[hidden])
Date: 2004-04-07 00:02:25

"John R. Bandela" <jbandela_at_[hidden]> writes:

> Thanks for the e-mail.
> The problem the char_separators not giving you the correct words and
> counts is a bug. It was caused by the latest changes to
> token_functions to speed up tokenizing non-input iterators. The
> version in the boost 1.31 should work (it is the one prior to the
> changes). I have also just fixed it in the CVS.

Thanks! All the docs still show char_delimiters_separator as the
default, though.

> Having char_delimiters_separator as the default tokenizing function
> was unintentional (ie it never got changed).


> As to simplifying the interface, I would really like to hear your ideas.

1. Allow "boost::use_default" for any of the template parameters

2. Supply a templated 1-argument ctor that constructs the start
   iterator from the argument and default constructs the end iterator

then I could write:

> t(std::cin);

> As to why char_delimiters_separator was deprecated, char_delimiters

What's that? It's not in the doc. Do you mean char_separator?

> was supposed to be a replacement for it, doing everything it did,
> but providing the user with more control in dealing with empty
> tokens. However, as you have brought up, it also has the unfortunate
> problem that in its default construction of returning punctuation as
> well as words and thus is not as simple to use.

Why not just change that.

> Perhaps the solution would be making a separate words_delimiter that
> is hard-wired to return only words (ie characters separated by
> either a space or a punctuation mark) and making that the default.

That'd be OK with me.

> As to why the mention of std::isspace and ispunct, I needed a default
> behavior when the user does not want to provide all the space and
> punctuation (it can be a somewhat long string). I also did not what to
> hard-code it as a string so I used those functions instead.

That doesn't demystify things for me. If you have an internal way to
use functions to determine kept and dropped, why don't you give me a
way to supply functions, too?

Dave Abrahams
Boost Consulting

Boost list run by bdawes at, gregod at, cpdaniel at, john at