Re: [Boost-Users] Tokenizer question

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2002-02-04 21:45:08

Hi Edward,

Several others have also found the return_delims parameter confusing, and
I've proposed that we remove that parameter.

John Bandela, would you mind if I go ahead and make that change?

On 2/4/02 2:03 PM, "Edward Diener" <eddielee_at_[hidden]> wrote:
> Two tokenizer questions
> 
> 1) I do not understand the constructor parameters to the
> char_delimits_separator class which is the default TokenizerFunction
> model. I assume the constructor would enable one to specify the
> delimiters for each token. I assume that the "returnable" parameter is a
> string literal of delimiters which are also returned as tokens and that
> "nonreturnable" is a string literal of delimiters which are not returned
> as tokens. I assume that the intersection of these string literals
> consists of all the delimiters. Given these assumptions, what is the
> purpose of the bool "return_delims" first parameter ? If "returnable"
> specifies a non-empty string literal, then why shouldn't these be the
> delimiters returned as tokens irregardless of the setting of
> "return_delims". If "returnable" specifies an empty string literal, then
> no delimiters will be returned as tokens once again irregardless of the
> setting of "return_delims". Would someone please explain why
> "return_delims" exists as a parameter in conjunction with the meaning of
> the "returnable" parameter ? It seems utterly redundant and unnecessary.
> 
> 2) How does one create one own's TokenizerFunction to be plugged into
> the tokenizer class template as the first template type ? The
> documentation on TokenizerFunction concept leaves me utterly confused on
> a practical level. I assume I am creating a template class here despite
> the name which suggests it is a function, but the documentation never
> explains what it is. Maybe someone from Boost or John Bandela should
> consider updating it.

John, will you address this?

Cheers,
Jeremy

-- 
Jeremy Siek                          http://www.osl.iu.edu/~jsiek
Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
C++ Booster (http://www.boost.org)   office phone: [phone]