Re: [boost] token_iterator comments, suggestions

From: Pavol Droba (droba_at_[hidden])
Date: 2004-06-05 16:40:12

Hi,

On Sat, Jun 05, 2004 at 08:16:17AM -0400, Gennadiy Rozental wrote:
> Hi,
> 
> For long time in my daytime projects I was using my class token_iterator
> designed based on old iterator_adaptor design (adopted for old compilers).
> Now when need arose for such functionality in Boost.Test. I looked in
> direction of boost::token_iterator. After some struggle I end up adopting my
> version to new design. Here some of comments and issues  that made me opt
> so.
> 

[snip] 

This might seem a little bit out of topic, but some of the points you have addressed
are already solved by find/split_iterator provided in the string algo library.
Maybe you can have a look there. Unfortunately, the documentation is not 100% ready yet.

find_iterator is also based on generic iterator concept, however, the iterator is the only
template parameter, so you can easily specialize for string.

Simple usage looks like this:

#include <boost/algorithm/string.hpp>

using namespace boost;
using namespace std;

....

typedef split_iterator<string::iterator> splitter_t;

{
   string str("hello;how are you,bye");

   for(splitter_t it=splitter_t(str, token_finder(is_any_of(";,")));
       it!=splitter_t();
       ++it)
   {
      cout << copy_iterator_range<string>(*it) << endl;
   }
}

I hope, that there are not too many syntax errors in the example.

What can be seen there:

- find/split_iterator is templated only by an iterator.

- Finding algorithm is provided at runtime in the form of a "finder".
  There is a coupe of those defined in the string_algo lib. Specification 
  can be seen here: 
    http://tinyurl.com/29dg3

- An example here uses token_finder. It takes a predicate that recognizes valid
  tokens (separators). Thare is also a bunch of classification predicates there
  to help here (boost/algorithm/string/classification.hpp)

- Default initialized find_iterator is an end mark for all iterations.
  Once the iterator reaches the end position, it will not assert on later increments,
  rather it will not change.

- A value of the iterator is an iterator_range. It a a pair of iterators delimiting
  begin and end of match.
  
Just have a look if you are interested.

Regards,

Pavol.