Tokenizer question

From: Edward Diener (eddielee_at_[hidden])
Date: 2002-02-04 14:03:32

Two tokenizer questions

1) I do not understand the constructor parameters to the 
char_delimits_separator class which is the default TokenizerFunction 
model. I assume the constructor would enable one to specify the 
delimiters for each token. I assume that the "returnable" parameter is a 
string literal of delimiters which are also returned as tokens and that 
"nonreturnable" is a string literal of delimiters which are not returned 
as tokens. I assume that the intersection of these string literals 
consists of all the delimiters. Given these assumptions, what is the 
purpose of the bool "return_delims" first parameter ? If "returnable" 
specifies a non-empty string literal, then why shouldn't these be the 
delimiters returned as tokens irregardless of the setting of 
"return_delims". If "returnable" specifies an empty string literal, then 
no delimiters will be returned as tokens once again irregardless of the 
setting of "return_delims". Would someone please explain why 
"return_delims" exists as a parameter in conjunction with the meaning of 
the "returnable" parameter ? It seems utterly redundant and unnecessary.

2) How does one create one own's TokenizerFunction to be plugged into 
the tokenizer class template as the first template type ? The 
documentation on TokenizerFunction concept leaves me utterly confused on 
a practical level. I assume I am creating a template class here despite 
the name which suggests it is a function, but the documentation never 
explains what it is. Maybe someone from Boost or John Bandela should 
consider updating it.