Re: [boost] Re: [tokenizer and token_iterator] request forpolicyorredesign

From: Pavol Droba (droba_at_[hidden])
Date: 2004-09-28 17:28:54

Hi,

On Tue, Sep 28, 2004 at 04:51:51PM -0400, Gennadiy Rozental wrote:
 
> This was the second goal, I was trying to achieve: best possible usability.
> If you try to combine these two slightly different functionalities under the
> same hood, you end up with interface that depends on iterator type:
> 
> token_iterator<std::string::iterator> it( str );
> 
> I see it as unacceptable burden (and that is why I do not like StringAlgo
> lib solution, but I plan to get back to that conversation later - I have
> saved message I need to answer). IMO the only acceptable template parameter
> should be character type, which in turn should be typedefed like STL do. IOW
> my interface of choice is:
> 
> token_iterator tit( str );
>

You will need to specify seperator, right? So you will need at least one more
parameter.

 
> Token iterator construction based on two iterators IMO is useful only for
> istream tokenization. I do provide this interface but as stand alone class.
> 

Just a few words about the StringAlgo solution. I remember our discussion,
be AFAIR we didn't come to a conclusion.

So here are few proposed changes as well as the current status of the things.

First of all, it is quite easy to get rid of the template parameters you don't like:

typedef boost::find_iterator<std::string::iterator> string_find_iterator;
typedef boost::find_iterator<std::wstring::iterator> wstring_find_iterator;

Two iterators are not needed for a long time already, find_iterators accept range
as a parameter.

Similary other parameters can be automated. (for instance, I can implement, that
a set of separators will be accepted and token_finder will be 
automaticaly instantiated).

If I remember correctly your last complain was, that dereferencing the iterator
does not yield a string, so you cannot manipulate it easily. 

find_iterator will dereference to an iterator_range. There is plenty of goodies
in the Boost.Range library that help you to live with it. For instance, there
are comparison operators between arbitrary range types, and if you need
more, you can always convert the range to something else using the
copy_iterator_range function (maybe something shorter and easier to use can
be provided as well).

Please consider these options. Some of them are not currently available since,
the StringAlgo lib has not been merged yet with the Range lib (I was afraid 
to do it before release). Any other ideas are more then welcome.

Regards,

Pavol