Boost logo

Boost Users :

From: Edward Diener (eddielee_at_[hidden])
Date: 2002-02-04 14:03:32

Two tokenizer questions

1) I do not understand the constructor parameters to the
char_delimits_separator class which is the default TokenizerFunction
model. I assume the constructor would enable one to specify the
delimiters for each token. I assume that the "returnable" parameter is a
string literal of delimiters which are also returned as tokens and that
"nonreturnable" is a string literal of delimiters which are not returned
as tokens. I assume that the intersection of these string literals
consists of all the delimiters. Given these assumptions, what is the
purpose of the bool "return_delims" first parameter ? If "returnable"
specifies a non-empty string literal, then why shouldn't these be the
delimiters returned as tokens irregardless of the setting of
"return_delims". If "returnable" specifies an empty string literal, then
no delimiters will be returned as tokens once again irregardless of the
setting of "return_delims". Would someone please explain why
"return_delims" exists as a parameter in conjunction with the meaning of
the "returnable" parameter ? It seems utterly redundant and unnecessary.

2) How does one create one own's TokenizerFunction to be plugged into
the tokenizer class template as the first template type ? The
documentation on TokenizerFunction concept leaves me utterly confused on
a practical level. I assume I am creating a template class here despite
the name which suggests it is a function, but the documentation never
explains what it is. Maybe someone from Boost or John Bandela should
consider updating it.

Boost-users list run by williamkempf at, kalb at, bjorn.karlsson at, gregod at, wekempf at