[boost] Re: Tokenizer doc problems/confusion (?)

From: David Abrahams (dave_at_[hidden])
Date: 2004-04-07 00:02:25

"John R. Bandela" <jbandela_at_[hidden]> writes:

> Thanks for the e-mail.
>
> The problem the char_separators not giving you the correct words and
> counts is a bug. It was caused by the latest changes to
> token_functions to speed up tokenizing non-input iterators. The
> version in the boost 1.31 should work (it is the one prior to the
> changes). I have also just fixed it in the CVS.

Thanks!  All the docs still show char_delimiters_separator as the
default, though.

> Having char_delimiters_separator as the default tokenizing function
> was unintentional (ie it never got changed).

OK.

> As to simplifying the interface, I would really like to hear your ideas.

1. Allow "boost::use_default" for any of the template parameters

2. Supply a templated 1-argument ctor that constructs the start
   iterator from the argument and default constructs the end iterator

then I could write:

       boost::tokenizer<
           boost::use_default,
           std::istreambuf_iterator<char> 
       > t(std::cin);

> As to why char_delimiters_separator was deprecated, char_delimiters
                                                      ^^^^^^^^^^^^^^^

What's that?  It's not in the doc.  Do you mean char_separator?

> was supposed to be a replacement for it, doing everything it did,
> but providing the user with more control in dealing with empty
> tokens. However, as you have brought up, it also has the unfortunate
> problem that in its default construction of returning punctuation as
> well as words and thus is not as simple to use. 

Why not just change that.

> Perhaps the solution would be making a separate words_delimiter that
> is hard-wired to return only words (ie characters separated by
> either a space or a punctuation mark) and making that the default.

That'd be OK with me.

> As to why the mention of std::isspace and ispunct, I needed a default
> behavior when the user does not want to provide all the space and
> punctuation (it can be a somewhat long string). I also did not what to
> hard-code it as a string so I used those functions instead.

That doesn't demystify things for me.  If you have an internal way to
use functions to determine kept and dropped, why don't you give me a
way to supply functions, too?

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com