Re: [Boost-Users] Tokenizer question

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2002-02-05 22:46:19

Hi All,

This is what I've done to fix the confusing constructor of
char_delimiters_separator.

I basically wanted to add another less-confusing constructor to the class,
and document the old constructor as deprecated (don't want to break
existing code). However, because of the default arguments in the original
constructor, there would be bad interactions between the new constructor
and the old.

Therefore, I instead just created a new class named char_separator and put
a note in the documention for char_delimiters_separator that the whole
class is deprecated.

I've checked in the code, tests, examples, and documentation for
char_separator. Let me know what you think.

Cheers,
Jeremy

P.S. It is still a bit annoying that it takes several lines to create a
tokenizer. When I get the time I'll had type generators and helper
functions (a la the iterator adaptor library) to make it easier to create
tokenizers (or if anyone else has the time, go for it!).

On Mon, 4 Feb 2002, Jeremy Siek wrote:

jsiek> Hi Edward,
jsiek>
jsiek> Several others have also found the return_delims parameter confusing, and
jsiek> I've proposed that we remove that parameter.
jsiek>
jsiek> John Bandela, would you mind if I go ahead and make that change?
jsiek>
jsiek> On 2/4/02 2:03 PM, "Edward Diener" <eddielee_at_[hidden]> wrote:
jsiek> > Two tokenizer questions
jsiek> >
jsiek> > 1) I do not understand the constructor parameters to the
jsiek> > char_delimits_separator class which is the default TokenizerFunction
jsiek> > model. I assume the constructor would enable one to specify the
jsiek> > delimiters for each token. I assume that the "returnable" parameter is a
jsiek> > string literal of delimiters which are also returned as tokens and that
jsiek> > "nonreturnable" is a string literal of delimiters which are not returned
jsiek> > as tokens. I assume that the intersection of these string literals
jsiek> > consists of all the delimiters. Given these assumptions, what is the
jsiek> > purpose of the bool "return_delims" first parameter ? If "returnable"
jsiek> > specifies a non-empty string literal, then why shouldn't these be the
jsiek> > delimiters returned as tokens irregardless of the setting of
jsiek> > "return_delims". If "returnable" specifies an empty string literal, then
jsiek> > no delimiters will be returned as tokens once again irregardless of the
jsiek> > setting of "return_delims". Would someone please explain why
jsiek> > "return_delims" exists as a parameter in conjunction with the meaning of
jsiek> > the "returnable" parameter ? It seems utterly redundant and unnecessary.
jsiek> >
jsiek> > 2) How does one create one own's TokenizerFunction to be plugged into
jsiek> > the tokenizer class template as the first template type ? The
jsiek> > documentation on TokenizerFunction concept leaves me utterly confused on
jsiek> > a practical level. I assume I am creating a template class here despite
jsiek> > the name which suggests it is a function, but the documentation never
jsiek> > explains what it is. Maybe someone from Boost or John Bandela should
jsiek> > consider updating it.
jsiek>
jsiek> John, will you address this?
jsiek>
jsiek> Cheers,
jsiek> Jeremy
jsiek>
jsiek> --
jsiek> Jeremy Siek                          http://www.osl.iu.edu/~jsiek
jsiek> Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
jsiek> C++ Booster (http://www.boost.org)   office phone: [phone]
jsiek>
jsiek>
jsiek>
jsiek> Info: <http://www.boost.org>
jsiek> Wiki: <http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl>
jsiek> Unsubscribe: <mailto:boost-users-unsubscribe_at_[hidden]>
jsiek>
jsiek>
jsiek> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
jsiek>
jsiek>

----------------------------------------------------------------------
 Jeremy Siek                          http://php.indiana.edu/~jsiek/
 Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
 C++ Booster (http://www.boost.org)   office phone: [phone]
----------------------------------------------------------------------